Commit Graph
229 Commits
Author SHA1 Message Date
kevinveenbirkenbach 152a576957 Make PostgreSQL startup wait idempotent by replacing init log markers with container state and SQL readiness checks
https://chatgpt.com/share/697e4312-610c-800f-977c-2946fdea48f8
2026-01-31 18:59:40 +01:00
kevinveenbirkenbach 9e3f4b9187 svc-db-postgres: fix race condition during initial Postgres startup
Wait explicitly for the Docker entrypoint init phase to complete by
checking the log marker before running any SQL commands. Then verify
database readiness with a real SQL roundtrip instead of pg_isready.

This avoids a race condition where pg_isready reports success while
the temporary init server is already shutting down, which caused
intermittent failures during initial deployment (especially with
PostGIS enabled).

https://chatgpt.com/share/697e3a68-e56c-800f-8438-6edb2a956802
2026-01-31 18:22:24 +01:00
kevinveenbirkenbach 59b2d09eb7 fix(shellcheck): suppress false-positive for trap-invoked function
SC2329 is a known ShellCheck limitation for functions invoked via trap.

https://chatgpt.com/share/69755296-2d74-800f-8675-261a283e3782
2026-01-25 00:18:35 +01:00
kevinveenbirkenbach db2a9a6539 fix(shellcheck): address unused loop var and trap false positive
- Replace seq-based loop with brace expansion to avoid unused variable warning
- Use quoted trap handler so ShellCheck detects restore_hba invocation

https://chatgpt.com/share/69755296-2d74-800f-8675-261a283e3782
2026-01-25 00:15:27 +01:00
kevinveenbirkenbach 383a59e9a1 feat(svc-db-postgres): robust superuser password rotation via script with explicit auth verification
- replace Python helper with Bash script executed via ansible.script
- wait for postgres readiness inside container (pg_isready)
- explicitly verify password auth via temporary pg_hba override
- distinguish CHANGED / UNCHANGED / ALREADY OK states
- map script result deterministically to Ansible changed/failed logic
- remove legacy postgres_set_superuser_password.py

https://chatgpt.com/share/69754a74-9d18-800f-89a6-304b083fbe50
2026-01-24 23:40:47 +01:00
kevinveenbirkenbach 9b567a2f0e feat(postgres): ensure superuser password idempotently via helper script
Add a Python helper script to ensure the postgres superuser password inside
the Docker container without shell/SQL quoting issues.

The script:
- checks whether the password already works
- updates it only if necessary
- signals change state via exit codes (0 = unchanged, 2 = changed)

The role now calls the script via ansible.builtin.script and maps rc to
changed/failed correctly.

https://chatgpt.com/share/697528b6-79f4-800f-96c9-da993feff9b9
2026-01-24 21:16:47 +01:00
kevinveenbirkenbach 8b1ae03e77 fix(env): prevent dotenv_quote double-quoting in env.j2 templates
- Replace `"{{ ... | dotenv_quote }}"` with `{{ ... | dotenv_quote }}` across roles
- Update integration test to enforce pure Jinja + dotenv_quote and forbid wrapped quotes

https://chatgpt.com/share/6974e6c6-51b8-800f-80a1-a76f885cf57e
2026-01-24 16:35:27 +01:00
kevinveenbirkenbach 737adaa567 fix(shell): quote all password variables in shell tasks
Ensure all password usages in shell commands are safely quoted using the
`| quote` filter to prevent shell injection issues and test failures.

Affected roles:
- svc-db-postgres
- web-app-gitea
- web-app-mediawiki
- web-app-moodle
- web-app-shopware
- web-app-xwiki

https://chatgpt.com/share/6974dd33-9470-800f-b469-d31fabcd59e1
2026-01-24 15:54:29 +01:00
kevinveenbirkenbach fdb2683ddb test(env): enforce dotenv_quote + double quotes for password vars in env.j2 templates
https://chatgpt.com/share/6974cbac-023c-800f-8198-fd85df2aa7bc
2026-01-24 15:32:33 +01:00
kevinveenbirkenbach aa7d0c642c feat(dotenv): add dotenv_quote filter and enforce quoted password vars in env.j2
https://chatgpt.com/share/6974cbac-023c-800f-8198-fd85df2aa7bc
2026-01-24 15:22:19 +01:00
kevinveenbirkenbach b6c2dd01c8 fix(postgres): replace pg_isready with real TCP auth wait and add log dump on failure
- Replace pg_isready with psql TCP password authentication check
- Eliminate Postgres init race condition in CI
- Add rescue handler dumping last 200 container log lines on failure

https://chatgpt.com/share/696f4386-fbbc-800f-bdde-f1e6c4145c84
2026-01-20 11:38:15 +01:00
kevinveenbirkenbach 6c50831456 fix(postgres): avoid jinja/go-template collision in debug rescue block
Replace inline lookup('pipe', 'docker info -f {{.DockerRootDir}}') with a raw shell task
to prevent Jinja parsing errors ("unexpected '.'") during rescue debugging.

https://chatgpt.com/share/696f4386-fbbc-800f-bdde-f1e6c4145c84
2026-01-20 11:22:23 +01:00
kevinveenbirkenbach 7272c3d56c fix(postgres): escape docker inspect go-templates in rescue debug
- Replace jinja-colliding docker inspect formats with raw-escaped go-templates
- Keep POSTGRES_CONTAINER rendered by Ansible while preserving Docker templates
- Improve reliability and readability of rescue debug output

https://chatgpt.com/share/696f4386-fbbc-800f-bdde-f1e6c4145c84
2026-01-20 10:40:01 +01:00
kevinveenbirkenbach e10a000291 fix(ci,postgres): stabilize GitHub runner env, move docker to /mnt, add deep debug for role login
- Introduce deterministic RUNNING_ON_GITHUB / RUNNING_ON_ACT detection
- Move Docker data-root to /mnt on real GitHub runners only
- Switch POSTGRES_PASSWORD schema from bcrypt to alphanumeric (avoid $ interpolation)
- Add robust rescue/debug block for ALTER ROLE login step
  (docker inspect, logs, pg_isready, psql auth, hba, activity, roles)
- Make docker-on-mnt deterministic (no migration, root-dir assertion)

Fixes flaky CI runs and intermittent PostgreSQL authentication failures.

https://chatgpt.com/share/696f4386-fbbc-800f-bdde-f1e6c4145c84
2026-01-20 09:57:32 +01:00
kevinveenbirkenbach c4fcd2b634 Added meta data for MariaDB and Postgres 2026-01-18 21:28:03 +01:00
kevinveenbirkenbach 79a1780a89 Removed unnecessary volume definitions 2026-01-18 20:47:22 +01:00
kevinveenbirkenbach 34b94b7059 More explicit overwritting to harden postgres 2026-01-18 20:01:58 +01:00
kevinveenbirkenbach b71a3fa6cd Hopes this solves Ci/CD postgres bug 2026-01-18 19:52:26 +01:00
kevinveenbirkenbach 9b87089bb0 Assert application_id integrity in svc-db-postgres init and harden load_app debug output
Add a defensive assertion in svc-db-postgres init to ensure application_id scope
is not polluted by upstream loaders.

Also normalize load_app debug output formatting and add a no-op hardening step
to make variable intent explicit for future scope-guarding refactors.

https://chatgpt.com/share/696ce9ee-68ac-800f-9ea0-d73c285af9a8
2026-01-18 18:20:04 +01:00
kevinveenbirkenbach bf1b12a489 fix(postgres): bind exposed port to docker bind host instead of localhost
Expose Postgres via DOCKER_BIND_HOST instead of POSTGRES_LOCAL_HOST to ensure
the port is reachable from the infinito container when running Docker-in-Docker
in CI/CD environments.

This avoids binding the published port to the container loopback interface
(127.0.0.1), which is not reachable from sibling containers.

https://chatgpt.com/share/696ce9ee-68ac-800f-9ea0-d73c285af9a8
2026-01-18 15:10:30 +01:00
kevinveenbirkenbach 66aec3b577 refactor(docker-compose): generate volumes via compose_volumes filter
Replace legacy volumes*.yml.j2 includes with the new compose_volumes filter across roles.
Add YAML-safe serialization for Ansible/Jinja proxy types to avoid PyYAML representer errors.
Remove obsolete volumes templates and extend unit tests to enforce required database_volume behavior.

https://chatgpt.com/share/696c2722-fd30-800f-bf01-dfcd3cb7ba72
2026-01-18 01:19:31 +01:00
kevinveenbirkenbach 89cc377d38 refactor(rdbms): fix recursive variable resolution by introducing database_consumer_id
- Replace database_application_id with database_consumer_id
- Use explicit consumer context for database resolution
- Fix recursive Jinja variable loop in sys-svc-rdbms
- Decouple discourse database network from global database_type

https://chatgpt.com/share/696bd40d-ebfc-800f-9ba8-1e6be919a629
2026-01-17 19:25:12 +01:00
kevinveenbirkenbach c104867735 Changed License to Infinito.Nexus Community License (Non-Commercial) 2026-01-17 18:26:05 +01:00
kevinveenbirkenbach c83a432e19 Optimized OpenLDAP loading and python installation procedures 2026-01-12 15:36:18 +01:00
kevinveenbirkenbach 023d44fcac fix(svc-db-postgres): install psycopg2 via pip for Ansible interpreter
Replace distro-specific python-psycopg2 package with psycopg2-binary installed
via ansible.builtin.pip to ensure the PostgreSQL modules work with the active
Ansible Python interpreter (e.g. Infinito venv) across all distros.

https://chatgpt.com/share/6962ff13-afa0-800f-93a1-9b0d306d98e2
2026-01-11 02:38:18 +01:00
kevinveenbirkenbach dd729fd35d Introduced lifecycle into meta/main.yml roles 2026-01-06 14:42:52 +01:00
kevinveenbirkenbach fd54faff65 refactor(ansible): replace pacman installs with package
Migrate most roles from community.general.pacman to the generic package module for better portability. Keep pacman where it is required (system upgrades and Arch-specific tasks). Split VirtualBox setup into an Arch-only task file and add a warning when skipped on non-Arch systems. Update integration test examples accordingly.

https://chatgpt.com/share/6947ce9f-3228-800f-8365-2253b5ed9451
2025-12-21 11:40:10 +01:00
kevinveenbirkenbach 7123841fb4 style: apply ruff autoformat 2025-12-17 16:37:21 +01:00
kevinveenbirkenbach db68f16e19 Added vars wrapper for RDMBS 2025-12-04 10:58:05 +01:00
kevinveenbirkenbach 5999763b4e Optimized Spacing 2025-12-04 10:50:28 +01:00
kevinveenbirkenbach 27c399123b Optimized handler loading. Requirerd due to 2.18->2.20 update 2025-12-04 00:53:27 +01:00
kevinveenbirkenbach 379b1d420e Refactor MariaDB and PostgreSQL roles for Ansible 2.20 compatibility and Infinito.Nexus conventions
- Replace legacy docker_container-based MariaDB deployment with docker-compose based workflow
- Add custom Dockerfile and docker-compose templates for MariaDB
- Split MariaDB command into separate arguments to avoid entrypoint parsing errors
- Introduce MARIADB_CUSTOM_IMAGE and MARIADB_EXPOSE_LOCAL variables
- Add docker_compose_flush_handlers to ensure correct handler execution on first run
- Replace utils/once/finalize.yml with utils/once/flag.yml for new run-once semantics
- Align variable naming with Infinito.Nexus UPPERCASE conventions
- Fix PostgreSQL custom image variable name (POSTGRES_CUSTOM_IMAGE_NAME → POSTGRES_CUSTOM_IMAGE)
- Remove obsolete flush_handlers var injection in svc-db-postgres/tasks/main.yml
- General cleanup after migration from Ansible 2.18 → 2.20

Conversation reference:
https://chatgpt.com/share/69306c81-9934-800f-b317-f53a8f246a73
2025-12-03 18:00:30 +01:00
kevinveenbirkenbach 716ebef33b Refactor task includes and update variable handling for Ansible 2.20 migration
This commit updates multiple roles to ensure compatibility with Ansible 2.20.
Several include paths and task-loading mechanisms required adjustments,
as Ansible 2.20 applies stricter evaluation rules for complex Jinja expressions
and no longer resolves certain relative include paths the way Ansible 2.18 did.

Key changes:
- Replaced legacy once_finalize.yml and once_flag.yml with the new structure
  under tasks/utils/once/finalize.yml and tasks/utils/once/flag.yml.
- Updated all include_tasks statements to use 'path_join' with playbook_dir,
  ensuring deterministic and absolute file resolution across roles.
- Fixed all network helper includes by converting direct relative paths such as
  'roles/docker-compose/tasks/utils/network.yml' to proper Jinja-evaluated paths.
- Normalized MATOMO_* variable names for consistency with the updated variable
  scope behavior in Ansible 2.20.
- Removed deprecated patterns that were implicitly supported in Ansible 2.18
  but break under the more strict variable and path resolution model in 2.20.

These changes are part of the full migration step required to ensure the
infinito-nexus roles remain stable, deterministic, and forward-compatible with
Ansible 2.20.

Details of the discussion and reasoning can be found in this conversation:
https://chatgpt.com/share/69300a8d-24d4-800f-bec0-e895a695618a
2025-12-03 11:02:34 +01:00
kevinveenbirkenbach 94f97ed1f3 Refactor: Migrate deprecated Ansible facts to ansible_facts[] syntax
Why:
- Ansible 2.20+ deprecates INJECT_FACTS_AS_VARS and direct usage of top-level ansible_* facts.
- This change updates all affected roles and vars files to the new supported syntax.
- Ensures compatibility with upcoming Ansible 2.24 removal of implicit fact injection.

Conversation reference:
https://chatgpt.com/share/692f639b-1380-800f-9f18-732f7108e9e2
2025-12-02 23:09:46 +01:00
kevinveenbirkenbach 0756e9d06e Optimized database variables 2025-12-02 11:59:55 +01:00
kevinveenbirkenbach 8314d7e6a6 Refactor DB initialization handling: externalize database_init flag for svc-db-mariadb and svc-db-postgres, remove internal auto-detection, and pass explicit init signal from sys-stk-back-stateful. Improves clarity, avoids incorrect credential lookups, and stabilizes central DB setup.
Reference: ChatGPT conversation https://chatgpt.com/share/692def1c-a79c-800f-b7dd-35e1845424bf
2025-12-01 20:40:30 +01:00
kevinveenbirkenbach e09f561f0b Refactor run-once orchestration and bootstrap Mailu/Mastodon in a single deploy
- Replace legacy utils/run_once.yml with the new helpers utils/once_flag.yml and utils/once_finalize.yml
- Introduce utils/compose_up.yml to ensure docker-compose stacks are up and to flush handlers safely without coupling to run-once flags
- Migrate all affected roles (desk-*, dev-*, sys-ctl-*, sys-svc-*, web-app-*, web-svc-*, util-*) to the new run-once helpers
- Rework sys-svc-msmtp to auto-load Mailu once per deploy, check reachability, and reuse the running stack instead of requiring multiple playbook passes
- Adjust web-app-mailu to integrate cert deployment, handler flushing, and run-once handling so Mailu is fully initialized in a single deploy
- Improve Matomo, CDN, logout and CSP/health-check related roles to cooperate with the new compose_up / once_* pattern
- Simplify alarm/backup/timer/service orchestration (sys-ctl-alm-*, sys-bkp-provider, sys-timer-cln-bkps, etc.) by moving run-once logic into dedicated 01_core.yml files
- Update integration tests so utils/once_flag.yml and utils/once_finalize.yml are recognised as valid run-once providers, keeping the global run_once_* guarantees consistent
- Align frontend injection and service dependencies so Mastodon- and Mailu-related services can be brought up coherently within a single deployment cycle rather than several iterations
2025-12-01 13:30:50 +01:00
kevinveenbirkenbach 81304934d5 Removed another block 2025-12-01 05:42:26 +01:00
kevinveenbirkenbach f5efbce205 feat(shopware): migrate to single Shopware base image and split services (web/worker/scheduler/init)
• Introduce init container and runtime-ready Dockerfile (Alpine) installing php83-gd/intl/pdo_mysql
• Disable composer scripts in builder and ignore build-time ext reqs
• New docker-compose template (web/worker/scheduler/opensearch) + persistent volumes
• Use TRUSTED_PROXIES env; fix APP_URL formatting; set OPENSEARCH_HOST=opensearch
• Replace SHOPWARE_PHP_CONTAINER refs with SHOPWARE_WEB_CONTAINER in tasks
• Render and copy init.sh via volumes path
• Remove old nginx/php split and legacy DB env task
• Fix svc-db-postgres var: database_type now uses entity_name
https://chatgpt.com/share/6907fc58-7c28-800f-a993-c207f28859c9
2025-11-03 01:51:38 +01:00
kevinveenbirkenbach 8beda2d45d fix(svc-db-postgres): pin Postgres version to 17-3.5, add entity_name var, and dynamically resolve major version for dev package
- Changed default Docker image version from 'latest' to '17-3.5' in config
- Introduced entity_name var for consistent lookups
- Added POSTGRES_VERSION and POSTGRES_VERSION_MAJOR extraction
- Updated Dockerfile to install postgresql-server-dev-<major> with default fallback to 'all'
- Minor YAML formatting improvements

Ref: https://chatgpt.com/share/68de40b4-2eb8-800f-ab5b-11cc873c3604
2025-10-02 11:07:17 +02:00
kevinveenbirkenbach 426ba32c11 feat(services): add CPU/RAM/PIDs defaults for heavy roles and align service names
Add per-service resource overrides (cpus, mem_reservation, mem_limit, pids_limit) for ollama, mariadb, postgres, confluence, gitlab, jira, keycloak, nextcloud; light formatting fixes in wordpress.

Rename service keys from generic 'application/web' to concrete names (jira, confluence, gitlab, keycloak) and update compose templates accordingly.

Jira: introduce JIRA_STORAGE_PATH and switch mounts/README accordingly.

https://chatgpt.com/share/68d2d96c-9bf4-800f-bbec-d4f2c0051c06
2025-09-23 21:43:50 +02:00
kevinveenbirkenbach 5d1210d651 feat(ai): introduce dedicated AI roles and wiring; clean up legacy AI stack
• Add svc-ai category under roles and load it in constructor stage

• Create new 'svc-ai-ollama' role (vars, tasks, compose, meta, README) and dedicated network

• Refactor former AI stack into separate app roles: web-app-flowise and web-app-openwebui

• Add web-app-minio role; adjust config (no central DB), meta (fa-database, run_after), compose networks include, volume key

• Provide user-focused READMEs for Flowise, OpenWebUI, MinIO, Ollama

• Networks: add subnets for web-app-openwebui, web-app-flowise, web-app-minio; rename web-app-ai → svc-ai-ollama

• Ports: rename ai_* keys to web-app-openwebui / web-app-flowise; keep minio_api/minio_console

• Add group_vars/all/17_ai.yml (OLLAMA_BASE_LOCAL_URL, OLLAMA_LOCAL_ENABLED)

• Replace hardcoded include paths with path_join in multiple roles (svc-db-postgres, sys-service, sys-stk-front-proxy, sys-stk-full-stateful, sys-svc-webserver, web-svc-cdn, web-app-keycloak)

• Remove obsolete web-app-ai templates/vars/env; split Flowise into its own role

• Minor config cleanups (CSP flags to {}, central_database=false)

https://chatgpt.com/share/68d15cb8-cf18-800f-b853-78962f751f81
2025-09-22 18:40:20 +02:00
kevinveenbirkenbach d523629cdd Refactor docker-compose templates: replace {% include 'build.yml.j2' %} with lookup() + indent for proper YAML embedding. Also adjusted build.yml.j2 to remove leading spaces. See: https://chatgpt.com/share/68ce584a-a430-800f-8e2a-0f96884cc8d1 2025-09-20 09:31:49 +02:00
kevinveenbirkenbach a4a8061998 Refactor: unify Docker build config via build.yml.j2 include
Replaced duplicated inline build definitions in multiple docker-compose.yml.j2
templates with a shared include (roles/docker-container/templates/build.yml.j2).
This ensures consistent use of pull_policy: never and Dockerfile context across
services (Postgres, Bookwyrm, Bridgy Fed, Chess, Confluence, Jira, Moodle,
OpenProject, Pretix, Roulette Wheel, WordPress, XWiki, Simpleicons).

Conversation: https://chatgpt.com/share/68cd8f35-b764-800f-9b00-2c837103d2fb
2025-09-19 19:13:44 +02:00
kevinveenbirkenbach b7a7be4737 Fix XWiki automation bootstrap:
- Accept HTTP 302 (Distribution Wizard redirects) in REST readiness and extension checks
- Treat 302 as missing admin user during bootstrap
- Move superadmin password to xwiki.cfg (correct location)
- Disable automatic Distribution Wizard start in xwiki.properties
- Standardize run_once includes for postgres, cdn, and xwiki roles

See: https://chatgpt.com/share/68c3a67b-80b4-800f-8a90-ebdcd4abb86c
2025-09-12 06:50:24 +02:00
kevinveenbirkenbach 7791bd8c04 Implement filter checks: ensure all defined filters are used and remove dead code
Integration tests added/updated:
- tests/integration/test_filters_usage.py: AST-based detection of filter definitions (FilterModule.filters), robust Jinja detection ({{ ... }}, {% ... %}, {% filter ... %}), plus Python call tracking; fails if a filter is used only under tests/.
- tests/integration/test_filters_are_defined.py: inverse check — every filter used in .yml/.yaml/.j2/.jinja2/.tmpl must be defined locally. Scans only inside Jinja blocks and ignores pipes inside strings (e.g., lookup('pipe', "... | grep ... | awk ...")) to avoid false positives like trusted_hosts, woff/woff2, etc.

Bug fixes & robustness:
- Build regexes without %-string formatting to avoid ValueError from literal '%' in Jinja tags.
- Strip quoted strings in usage analysis so sed/grep/awk pipes are not miscounted as filters.
- Prevent self-matches in the defining file.

Cleanup / removal of dead code:
- Removed unused filter plugins and related unit tests:
  * filter_plugins/alias_domains_map.py
  * filter_plugins/get_application_id.py
  * filter_plugins/load_configuration.py
  * filter_plugins/safe.py
  * filter_plugins/safe_join.py
  * roles/svc-db-openldap/filter_plugins/build_ldap_nested_group_entries.py
  * roles/sys-ctl-bkp-docker-2-loc/filter_plugins/dict_to_cli_args.py
  * corresponding tests under tests/unit/*
- roles/svc-db-postgres/filter_plugins/split_postgres_connections.py: dropped no-longer-needed list_postgres_roles API; adjusted tests.

Misc:
- sys-stk-front-proxy/defaults/main.yml: clarified valid vhost_flavour values (comma-separated).

Ref: https://chatgpt.com/share/68b56bac-c4f8-800f-aeef-6708dbb44199
2025-09-01 11:47:51 +02:00
kevinveenbirkenbach 5f66c1a622 feat(postgres): add split_postgres_connections filter and average pool fact
Compute POSTGRES_ALLOWED_AVG_CONNECTIONS once and propagate to app roles (gitlab, mastodon, listmonk, matrix, pretix, mobilizon, openproject, discourse). Fix docker-compose postgres command (-c flags split). Add unit tests. Minor env/locale tweaks and includes.

Conversation: https://chatgpt.com/share/68b48e72-cc28-800f-9c21-270cbc17d82a
2025-08-31 20:04:14 +02:00
kevinveenbirkenbach 2fccebbd1f Enforce uppercase README.md and TODO.md filenames
- Renamed all Readme.md → README.md
- Renamed all Todo.md → TODO.md
- Added integration test (tests/integration/test_filename_conventions.py) to automatically check naming convention.

Background:
Consistency in file naming (uppercase README.md and TODO.md) avoids issues with case-sensitive filesystems and ensures desktop cards (e.g. Pretix) are properly included.
Ref: https://chatgpt.com/share/68b1d135-c688-800f-9441-46a3cbfee175
2025-08-29 18:11:53 +02:00
kevinveenbirkenbach dece6228a4 Refactor docker-compose build logic and pull policy
- Added conditional '--pull' flag on retry in docker-compose build handler, tied to MODE_UPDATE
- Added 'pull_policy: never' to multiple docker-compose service templates to prevent unwanted image pulls
- Fixed minor formatting issues (e.g. Nextcloud volume spacing, WordPress desktop alignment)

Reference: https://chatgpt.com/share/68b0207a-4d9c-800f-b76f-9515885e5183
2025-08-28 11:25:35 +02:00
kevinveenbirkenbach cb539b038c Marked as not changed 2025-08-15 19:00:03 +02:00