Decoupeld database, docker and proxy

This commit is contained in:
Kevin Veen-Birkenbach 2025-07-09 14:21:30 +02:00
parent 73329506a9
commit c9c73cbdb2
No known key found for this signature in database
GPG Key ID: 44D8F11FD62F878E
55 changed files with 386 additions and 103 deletions

184
roles/categories.yml Normal file
View File

@ -0,0 +1,184 @@
categories:
core:
title: "Core & System"
description: "Fundamental system configuration"
icon: "fas fa-cogs"
gen:
title: "gen-*"
description: "Helper roles & installers (git, locales, timer, etc.)"
icon: "fas fa-wrench"
desk:
title: "desk-*"
description: "Desktop environment roles & apps (GNOME, browser, LibreOffice, etc.)"
icon: "fas fa-desktop"
desk:
applications:
title: "Desktop Applications"
description: "Setup & utilities for desktop apps"
icon: "fas fa-desktop"
utils:
title: "utils-desk-*"
description: "Utility roles for desktop tools & development"
icon: "fas fa-tools"
srv:
web:
core:
title: "srv-web-core"
description: "Install & configure base Nginx server"
icon: "fas fa-server"
tls:
title: "srv-web-tls-*"
description: "Deploy & renew TLS certificates"
icon: "fas fa-lock"
proxy:
title: "srv-web-proxy-*"
description: "Proxy & vhost orchestration"
icon: "fas fa-project-diagram"
injector:
core:
title: "srv-web-injector-core"
description: "Inject core HTML modifiers"
icon: "fas fa-code"
css:
title: "srv-web-injector-css"
description: "Inject CSS into responses"
icon: "fas fa-paint-brush"
iframe:
title: "srv-web-injector-iframe"
description: "Inject iframe notifier"
icon: "fas fa-window-maximize"
javascript:
title: "srv-web-injector-javascript"
description: "Inject JS into responses"
icon: "fas fa-code"
matomo:
title: "srv-web-injector-matomo"
description: "Inject Matomo tracking code"
icon: "fas fa-chart-pie"
composer:
title: "srv-web-composer"
description: "Compose multiple filters into one include"
icon: "fas fa-layer-group"
web:
svc:
title: "web-svc-*"
description: "Static content servers (assets, HTML, legal, files)"
icon: "fas fa-file"
app:
title: "web-app-*"
description: "Deployable web applications (GitLab, Nextcloud, Mastodon, etc.)"
icon: "fas fa-docker"
net:
general:
title: "net-*"
description: "Network setup (DNS, Let's Encrypt HTTP, WireGuard, etc.)"
icon: "fas fa-globe"
svc:
title: "svc-*"
description: "Docker infrastructure services (DBMS, LDAP, Redis, etc.)"
icon: "fas fa-database"
wireguard:
core:
title: "net-wireguard-core"
description: "Core WireGuard configuration"
icon: "fas fa-network-wired"
firewalled:
title: "net-wireguard-firewalled"
description: "WireGuard with firewall rules"
icon: "fas fa-shield-alt"
plain:
title: "net-wireguard-plain"
description: "WireGuard without extra firewall"
icon: "fas fa-network-wired"
monitoring:
bot:
title: "mon-bot-*"
description: "Bot-style health checks (disk, Docker, webserver, etc.)"
icon: "fas fa-robot"
core:
title: "monitor-core-*"
description: "Low-level monitors (journalctl, containers, disk space, etc.)"
icon: "fas fa-chart-area"
alerting:
title: "Alerting"
description: "Notification handlers for system events"
icon: "fas fa-bell"
subcategories:
email:
title: "alert-email"
description: "Send alerts via email"
icon: "fas fa-envelope"
telegram:
title: "alert-telegram"
description: "Send alerts via Telegram"
icon: "fab fa-telegram-plane"
compose:
title: "alert-compose"
description: "Compose multiple alert handlers"
icon: "fas fa-project-diagram"
maintenance:
title: "Maintenance & Healing"
description: "Periodic maintenance & auto-recovery"
icon: "fas fa-tools"
subcategories:
general:
title: "maint-*"
description: "Periodic tasks (Btrfs balancing, swapfile, etc.)"
icon: "fas fa-sync-alt"
docker:
title: "maint-docker-*"
description: "Automated Docker recovery & restarts"
icon: "fas fa-docker"
cleanup:
title: "cln-*"
description: "Housekeeping tasks (backups, certs, logs, etc.)"
icon: "fas fa-broom"
backup:
title: "Backup & Restore"
description: "Backup strategies & restore procedures"
icon: "fas fa-hdd"
subcategories:
general:
title: "bkp-*"
description: "Local & remote backups (files, volumes, DBs)"
icon: "fas fa-cloud-upload-alt"
updates:
title: "Updates & Package Management"
description: "OS & package updates"
icon: "fas fa-sync"
subcategories:
os:
title: "update-*"
description: "Automatic OS & package updates (apt, Docker, pip, etc.)"
icon: "fas fa-download"
pkgmgr:
title: "pkgmgr-*"
description: "Language/platform package managers (npm, pip, AUR, etc.)"
icon: "fas fa-box-open"
users:
title: "Users & Access"
description: "User accounts & access control"
icon: "fas fa-users"
subcategories:
general:
title: "user-*"
description: "Create user accounts & SSH keys"
icon: "fas fa-user"
administrator:
title: "user-administrator"
description: "Config for admin users"
icon: "fas fa-user-shield"
root:
title: "user-root"
description: "Config for root user"
icon: "fas fa-user-shield"

View File

@ -0,0 +1,25 @@
# Database Docker Composition
This role combines the central RDBMS role (`svc-rdbms-central`) with Docker Compose to deliver a ready-to-use containerized database environment.
## Features
- **Central RDBMS Integration**
Includes the `svc-rdbms-central` role, which handles backups, restores, user and permission management for your relational database system (PostgreSQL, MariaDB, etc.).
- **Docker Compose**
Utilizes the standalone `docker-compose` role to define and bring up containers, networks, and volumes automatically.
- **Variable Load Order**
1. Docker Compose variables (`roles/docker-compose/vars/docker-compose.yml`)
2. Database variables (`roles/svc-rdbms-central/vars/database.yml`)
Ensures compose ports and volumes are defined before the database role consumes them.
The role will load both sub-roles and satisfy all dependencies transparently.
## Task Breakdown
1. **Set Fact** `database_application_id` to work around lazyloading ordering.
2. **Include Vars** in the specified order.
3. **Invoke** `docker-compose` role to create containers, networks, and volumes.
4. **Invoke** `svc-rdbms-central` role to provision the database, backups, and users.

View File

@ -0,0 +1,17 @@
galaxy_info:
author: "Kevin Veen-Birkenbach"
description: >
Combines Docker Compose with a central RDBMS role to automatically
provision database containers with backup, user, and permission management.
company: |
Kevin Veen-Birkenbach
Consulting & Coaching Solutions
https://www.veen.world
license: "CyMaIS NonCommercial License (CNCL)"
license_url: "https://s.veen.world/cncl"
galaxy_tags:
- docker
- database
- rdbms
- backup
- compose

View File

@ -0,0 +1,17 @@
- name: "set database_application_id (Needed due to lazzy loading issue)"
set_fact:
database_application_id: "{{ application_id }}"
- name: "Load database variables"
include_vars: "{{ item }}"
loop:
- "{{ cmp_db_docker_vars_file_docker }}" # Important to load docker variables first so that database can use them
- "{{ cmp_db_docker_vars_file_db }}" # Important to load them before docker role so that backup can use them
- name: "Load docker-compose for {{ application_id }}"
include_role:
name: docker-compose
- name: "Load central rdbms for {{ application_id }}"
include_role:
name: svc-rdbms-central

View File

@ -0,0 +1,2 @@
cmp_db_docker_vars_file_db: "{{ playbook_dir }}/roles/svc-rdbms-central/vars/database.yml"
cmp_db_docker_vars_file_docker: "{{ playbook_dir }}/roles/docker-compose/vars/docker-compose.yml"

View File

@ -25,5 +25,4 @@ galaxy_info:
issue_tracker_url: https://s.veen.world/cymaisissues
documentation: https://s.veen.world/cymais
dependencies:
- srv-web-proxy-core
- docker-container # Necessary for template use

View File

@ -40,4 +40,5 @@
register: docker_ps
changed_when: (docker_ps.stdout | trim) == ""
notify: docker compose up
when: not (docker_compose_template.changed or env_template.changed)
when: not (docker_compose_template.changed or env_template.changed)
ignore_errors: true

View File

@ -0,0 +1,2 @@
dependencies:
- docker-core

View File

@ -24,5 +24,5 @@ galaxy_info:
issue_tracker_url: https://s.veen.world/cymaisissues
documentation: https://s.veen.world/cymais
dependencies:
- docker
- srv-web-https
- srv-web-https
- srv-web-core

View File

@ -24,4 +24,4 @@ galaxy_info:
issue_tracker_url: https://s.veen.world/cymaisissues
documentation: https://s.veen.world/cymais
dependencies:
- srv-web-core
- srv-web-proxy-core

View File

@ -1,4 +1,5 @@
---
- name: "include docker-compose role"
include_role:
name: docker-compose
@ -23,6 +24,14 @@
ipam_config:
- subnet: "{{ networks.local.central_ldap.subnet }}"
- name: "Wait for LDAP to be available"
wait_for:
host: "127.0.0.1"
port: "{{ ports.localhost.ldap.openldap }}"
delay: 5
timeout: 120
state: started
- name: "Reset LDAP admin passwords"
include_tasks: reset_admin_passwords.yml
when: applications[application_id].network.local

View File

@ -1,18 +1,3 @@
- name: "set database_application_id (Needed due to lazzy loading issue)"
set_fact:
database_application_id: "{{ application_id }}"
- name: "Load database variables"
include_vars: "{{ item }}"
loop:
- "{{ docker_var_file }}" # Important to load docker variables first so that database can use them
- "{{ database_var_file }}" # Important to load them before docker role so that backup can use them
# Docker Routines
- name: "Include docker-compose role"
include_role:
name: docker-compose
# The following env file will just be used from the dedicated mariadb container
# and not the {{applications.mariadb.hostname }}-database
- name: "Create {{database_env}}"

View File

@ -1,2 +0,0 @@
database_var_file: "{{ role_path }}/vars/database.yml"
docker_var_file: "{{playbook_dir}}/roles/docker-compose/vars/docker-compose.yml"

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role for {{application_id}} to receive certs & do modification routines"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -10,9 +10,9 @@
state: present
when: run_once_docker_discourse is not defined
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
when: run_once_docker_discourse is not defined
- name: "include role srv-web-proxy-domain for {{application_id}}"

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
include_role:
name: svc-rdbms-central
- name: "load docker and db for {{application_id}}"
include_role:
name: cmp-db-docker
- name: "Include setup for domain '{{ domain }}'"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "create {{ friendica_host_ldap_config }}"
template:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,2 +1 @@
dependencies:
- srv-web-proxy-core

View File

@ -1,3 +1,7 @@
- name: "load docker and db for {{application_id}}"
include_role:
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:
name: srv-web-proxy-domain
@ -5,15 +9,3 @@
domain: "{{ domains | get_domain(application_id) }}"
http_port: "{{ ports.localhost.http[application_id] }}"
- name: "docker jenkins"
docker_compose:
application_id: jenkins
definition:
application:
image: jenkins/jenkins:lts
restart: "{{docker_restart_policy}}"
ports:
- "127.0.0.1:{{ports.localhost.http[application_id]}}:8080"
volumes:
- jenkins_data:/var/jenkins_home
log_driver: journald

View File

@ -0,0 +1,10 @@
{% include 'roles/docker-compose/templates/base.yml.j2' %}
application:
image: jenkins/jenkins:lts
restart: "{{docker_restart_policy}}"
ports:
- "127.0.0.1:{{ports.localhost.http[application_id]}}:8080"
volumes:
- jenkins_data:/var/jenkins_home
log_driver: journald
{% include 'roles/docker-compose/templates/networks.yml.j2' %}

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -0,0 +1,42 @@
# Draft
- name: Wait until Keycloak is up
uri:
url: "{{ keycloak_server_host_url }}/realms/{{ keycloak_realm }}"
method: GET
status_code: 200
validate_certs: false
register: keycloak_up
retries: 30
delay: 5
until: keycloak_up.status == 200
- name: Log in with kcadm.sh
shell: |
{{ keycloak_kcadm_path }} config credentials \
--server {{ keycloak_server_internal_url }} \
--realm master \
--user {{ keycloak_administrator_username }} \
--password {{ keycloak_administrator_password }}
- name: Retrieve LDAP component ID
shell: |
{{ keycloak_kcadm_path }} get components \
-r {{ keycloak_realm }} \
--query 'providerId=ldap' \
--fields id \
--format json \
| jq -r '.[0].id'
register: ldap_component
changed_when: false
- name: Update LDAP bind password
vars:
new_bind_password: "MyNewLdapPassword123!"
shell: |
{{ keycloak_kcadm_path }} update components/{{ ldap_component.stdout }} \
-r {{ keycloak_realm }} \
-s 'config.bindCredential=["{{ new_bind_password }}"]'
no_log: true
register: update_bind
changed_when: update_bind.rc == 0

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: Set nginx_docker_reverse_proxy_extra_configuration based on applications[application_id].public_api_activated
set_fact:

View File

@ -1,7 +1,7 @@
---
- name: "Include svc-rdbms-central"
include_role:
name: svc-rdbms-central
- name: "load docker and db for {{application_id}}"
include_role:
name: cmp-db-docker
when: run_once_docker_mailu is not defined
- name: "Include role srv-web-proxy-domain for {{ application_id }}"

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "Include setup for domain '{{ domain }}'"
include_role:

View File

@ -1,6 +1,6 @@
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -8,9 +8,9 @@
bridges: "{{ bridges_configuration | filter_enabled_bridges(applications[application_id].plugins) }}"
changed_when: false
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role for {{application_id}} to receive certs & do modification routines"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include tasks setup-domain.yml with {{domains | get_domain(application_id)}}"
include_tasks: setup-domain.yml

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "create {{ nextcloud_host_config_additives_directory }}"
file:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include create-domains.yml for peertube"
include_tasks: create-domains.yml

View File

@ -1,5 +1,5 @@
- name: "load variables from {{ database_var_file }}"
include_vars: "{{ database_var_file }}"
- name: "load variables from {{ cmp_db_docker_vars_file_db }}"
include_vars: "{{ cmp_db_docker_vars_file_db }}"
- name: "loading database configuration variables"
include_vars:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role for {{application_id}} to receive certs & do modification routines"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -1,7 +1,7 @@
---
- name: "Include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "Include role srv-web-proxy-domain for {{ application_id }}"
include_role:
@ -21,7 +21,7 @@
- name: "Transfering msmtprc to {{ host_msmtp_conf }}"
template:
src: "{{ playbook_dir }}/roles/msmtp/templates/msmtprc.conf.j2"
src: "{{ playbook_dir }}/roles/gen-msmtp/templates/msmtprc.conf.j2"
dest: "{{ host_msmtp_conf }}"
notify: docker compose up

View File

@ -1,7 +1,7 @@
---
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
- name: "include role srv-web-proxy-domain for {{application_id}}"
include_role:

View File

@ -0,0 +1 @@
# General Construction Tasks

View File

@ -3,9 +3,9 @@
{% if database_type | bool %}
{% raw %}
- name: "include svc-rdbms-central"
- name: "load docker and db for {{application_id}}"
include_role:
name: svc-rdbms-central
name: cmp-db-docker
when: run_once_docker_{% endraw %}{{ application_id }}{% raw %} is not defined
{% endraw %}