implemented check for anonymous volumes

This commit is contained in:
Kevin Veen-Birkenbach 2023-11-17 16:53:56 +01:00
parent 66280fdbde
commit a519a09725
24 changed files with 177 additions and 42 deletions

View File

@ -32,7 +32,7 @@ This software allows to setup the docker following applications:
This software shipts the following tools which are natively setup on the server:
- [Backups Cleanup](./roles/cleanup-backups-timer/README.md) - Cleans up old backups
- [Btrfs Health Check](./roles/health-btrfs/README.md) - Checks the health of Btrfs file systems
- [Docker Health Check](./roles/health-docker/) - Checks the health of docker containers
- [Docker Health Check](./roles/health-docker-container/) - Checks the health of docker containers
- [Docker Reverse Proxy](./roles/docker-reverse-proxy/README.md) - Docker Reverse Proxy Solution
- [Docker Volume Backup](./roles/backup-docker-to-local/) - Backup Solution for Docker Volumes
- [Pull Primary Backups](./roles/backup-remote-to-local/README.md) - Pulls the backups from another server and stores them

View File

@ -7,8 +7,9 @@ randomized_delay_sec: "15min"
on_calendar_health_btrfs: "*-*-* 00:00:00"
on_calendar_health_journalctl: "*-*-* 00:00:00"
on_calendar_health_disc_space: "*-*-* 06,12,18,00:00:00"
on_calendar_health_docker: "*-*-* 09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,00,01,02:00:00"
on_calendar_health_nginx: "*-*-* 09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,00,01,02:15:00"
on_calendar_health_docker_container: "*-*-* 09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,00,01,02:00:00"
on_calendar_health_docker_volumes: "*-*-* 09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,00,01,02:15:00"
on_calendar_health_nginx: "*-*-* 09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,00,01,02:45:00"
on_calendar_cleanup_backups: "*-*-* 06,12,18,00:30:00"
on_calendar_cleanup_disc_space: "*-*-* 07,13,19,01:30:00"

View File

@ -18,6 +18,8 @@ services:
- database
depends_on:
- database
volumes:
- data:/var/www/html
database:
logging:
driver: journald
@ -38,6 +40,7 @@ services:
retries: 5
volumes:
database:
data:
networks:
default:
driver: bridge

View File

@ -1,5 +1,6 @@
dependencies:
- backup-docker-to-local
- user-administrator
- health-docker
- health-docker-container
- health-docker-volumes
- heal-docker

View File

@ -0,0 +1,21 @@
# Health Check for Docker Containers
## Description
This Ansible role is designed to ensure the health of Docker containers running on a system. It includes a script that checks for unhealthy or exited Docker containers and sets up a systemd service and timer to regularly execute this check.
## Files
- `vars/main.yml`: Variable definitions for the script's directory.
- `handlers/main.yml`: Handlers to reload and restart the systemd service and timer.
- `files/health-docker-container.sh`: The script that checks the container health.
- `tasks/main.yml`: Tasks to create necessary directories, copy scripts, and create systemd service and timer.
- `templates/health-docker-container.service.j2`: Systemd service template.
- `templates/health-docker-container.timer.j2`: Systemd timer template.
- `meta/main.yml`: Meta information declaring dependencies for the role.
## Usage
To use this role, include it in your playbook and set the `path_administrator_scripts` variable to the desired path for the health check scripts.
Ensure that the `systemd_notifier` dependency is satisfied for error notifications.

View File

@ -0,0 +1,11 @@
- name: "reload health-docker-container.service"
systemd:
name: health-docker-container.service
enabled: yes
daemon_reload: yes
- name: "restart health-docker-container.timer"
systemd:
name: health-docker-container.timer
state: restarted
enabled: yes
daemon_reload: yes

View File

@ -0,0 +1,22 @@
- name: "create {{health_docker_container_folder}}"
file:
path: "{{health_docker_container_folder}}"
state: directory
mode: 0755
- name: create health-docker-container.sh
copy:
src: health-docker-container.sh
dest: "{{health_docker_container_folder}}health-docker-container.sh"
- name: create health-docker-container.service
template: src=health-docker-container.service.j2 dest=/etc/systemd/system/health-docker-container.service
notify: reload health-docker-container.service
- name: create health-docker-container.timer
template:
src: health-docker-container.timer.j2
dest: "/etc/systemd/system/health-docker-container.timer"
register: health_docker_container_timer
changed_when: health_docker_container_timer.changed or activate_all_timers | default(false) | bool
notify: restart health-docker-container.timer

View File

@ -0,0 +1,7 @@
[Unit]
Description=Checking docker health
OnFailure=systemd-notifier@%n.service
[Service]
Type=oneshot
ExecStart=/bin/bash {{health_docker_container_folder}}health-docker-container.sh

View File

@ -0,0 +1,10 @@
[Unit]
Description=starts health-docker-container.service
[Timer]
OnCalendar={{on_calendar_health_docker_container}}
RandomizedDelaySec={{randomized_delay_sec}}
Persistent=false
[Install]
WantedBy=timers.target

View File

@ -0,0 +1 @@
health_docker_container_folder: "{{path_administrator_scripts}}health-docker-container/"

View File

@ -0,0 +1,24 @@
# Health Check for Docker Volumes
## Description
This role checks for anonymous Docker volumes that are not bound to a container and may be left over from previous operations. It provides a cleanup mechanism by identifying such volumes and possibly taking action against them.
## Files
- `vars/main.yml`: Variable definitions for the script's directory.
- `handlers/main.yml`: Handlers to reload and restart the systemd service and timer.
- `files/health-docker-volumes.sh`: The script that checks for anonymous Docker volumes.
- `tasks/main.yml`: Tasks to create necessary directories, copy scripts, and create systemd service and timer.
- `templates/health-docker-volumes.service.j2`: Systemd service template.
- `templates/health-docker-volumes.timer.j2`: Systemd timer template.
- `meta/main.yml`: Meta information declaring dependencies for the role.
## Usage
This role can be included in your playbook. Set the `path_administrator_scripts` variable to determine where the health check scripts should reside.
The role uses `systemd_notifier` for failure notifications, so ensure this dependency is present in your environment.
## Created with AI
This script was created with the help of AI. The full conversation you find [here](https://chat.openai.com/share/1fa829f1-f001-4111-b1d4-1b2e3d583da2).

View File

@ -0,0 +1,32 @@
#!/bin/bash
anonymous_volumes=$(docker volume ls --format "{{.Name}}" | grep -E '^[a-f0-9]{64}$')
if [ -z "$anonymous_volumes" ]; then
echo "No anonymous volumes found."
exit 0
fi
echo "Anonymous volumes found:"
for volume in $anonymous_volumes; do
container_ids=$(docker ps -aq --filter volume=$volume)
if [ -z "$container_ids" ]; then
echo "Volume $volume is not used by any running containers."
continue
fi
for container_id in $container_ids; do
container_name=$(docker inspect --format '{{ .Name }}' $container_id | sed 's#^/##')
mount_path=$(docker inspect --format "{{ range .Mounts }}{{ if eq .Name \"$volume\" }}{{ .Destination }}{{ end }}{{ end }}" $container_id)
if [ -n "$mount_path" ]; then
echo "Volume $volume is used by container $container_name at mount path $mount_path"
else
echo "Volume $volume is used by container $container_name, but mount path could not be determined."
fi
done
done
exit 1

View File

@ -0,0 +1,11 @@
- name: "reload health-docker-volumes.service"
systemd:
name: health-docker-volumes.service
enabled: yes
daemon_reload: yes
- name: "restart health-docker-volumes.timer"
systemd:
name: health-docker-volumes.timer
state: restarted
enabled: yes
daemon_reload: yes

View File

@ -0,0 +1,2 @@
dependencies:
- systemd_notifier

View File

@ -0,0 +1,22 @@
- name: "create {{health_docker_volumes_folder}}"
file:
path: "{{health_docker_volumes_folder}}"
state: directory
mode: 0755
- name: create health-docker-volumes.sh
copy:
src: health-docker-volumes.sh
dest: "{{health_docker_volumes_folder}}health-docker-volumes.sh"
- name: create health-docker-volumes.service
template: src=health-docker-volumes.service.j2 dest=/etc/systemd/system/health-docker-volumes.service
notify: reload health-docker-volumes.service
- name: create health-docker-volumes.timer
template:
src: health-docker-volumes.timer.j2
dest: "/etc/systemd/system/health-docker-volumes.timer"
register: health_docker_volumes_timer
changed_when: health_docker_volumes_timer.changed or activate_all_timers | default(false) | bool
notify: restart health-docker-volumes.timer

View File

@ -4,4 +4,4 @@ OnFailure=systemd-notifier@%n.service
[Service]
Type=oneshot
ExecStart=/bin/bash {{health_docker_folder}}health-docker.sh
ExecStart=/bin/bash {{health_docker_volumes_folder}}health-docker-volumes.sh

View File

@ -1,8 +1,8 @@
[Unit]
Description=starts health-docker.service
Description=starts health-docker-volumes.service
[Timer]
OnCalendar={{on_calendar_health_docker}}
OnCalendar={{on_calendar_health_docker_volumes}}
RandomizedDelaySec={{randomized_delay_sec}}
Persistent=false

View File

@ -0,0 +1 @@
health_docker_volumes_folder: "{{path_administrator_scripts}}health-docker-volumes/"

View File

@ -1,11 +0,0 @@
- name: "reload health-docker.service"
systemd:
name: health-docker.service
enabled: yes
daemon_reload: yes
- name: "restart health-docker.timer"
systemd:
name: health-docker.timer
state: restarted
enabled: yes
daemon_reload: yes

View File

@ -1,22 +0,0 @@
- name: "create {{health_docker_folder}}"
file:
path: "{{health_docker_folder}}"
state: directory
mode: 0755
- name: create health-docker.sh
copy:
src: health-docker.sh
dest: "{{health_docker_folder}}health-docker.sh"
- name: create health-docker.service
template: src=health-docker.service.j2 dest=/etc/systemd/system/health-docker.service
notify: reload health-docker.service
- name: create health-docker.timer
template:
src: health-docker.timer.j2
dest: "/etc/systemd/system/health-docker.timer"
register: health_docker_timer
changed_when: health_docker_timer.changed or activate_all_timers | default(false) | bool
notify: restart health-docker.timer

View File

@ -1 +0,0 @@
health_docker_folder: "{{path_administrator_scripts}}health-docker/"

View File

@ -9,7 +9,7 @@ server
location /
{
root /usr/share/nginx/homepage;
root {{nginx_homepage_root}};
index index.html index.htm;
}
}