Merge branch 'feature/updater-lockdown-probe' into feature/updater

Brings the Nextcloud plugin-phase speed-up (config set in one occ boot, plugin installs without the redundant enable) and the occ batch error labels, Talk summaries only with a speech-to-text provider, trusted_proxies through occ only, the 061 criteria for signaling and recording, the ERPNext Server Scripts switch services.erpnext.server_scripts, the Bluesky broker's recovery from a rotated encryption key, the custom-role copy that skips .claude, and two main merges. Quality-high pass 16 ran green on the merged tip.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5.5 committed 2026-10-30 09:43:47 +01:00
commit 8c14873225
45 files changed
+821 -138

No files matched your search

+1 -1
View File
@@ -4,7 +4,7 @@
- Implement rbac administration interface
- Enable IP6 for docker.
- backup docker to local für ca optimieren
- Drop the `php_memory_unlimited` mount ([memory-unlimited.ini](roles/sys-svc-compose-php/files/memory-unlimited.ini), `memory_limit = -1`) from each role whose image ships PHP 8.2.35, 8.3.36, 8.4.27, 8.5.12 or later, and the file, its install task and `FILE_PHP_MEMORY_UNLIMITED_INI` once no role mounts it. It is the interim mitigation for CVE-2026-109618 and overrides the per-role `service_memory_mb` share, so a single request can exhaust the container instead of hitting a PHP limit. Still mounted by `web-app-magento` (`markoshust/magento-php:8.4-fpm`, PHP 8.4.21), `web-app-mailu` (`ghcr.io/heviat/webmail:2024.06`, the OIDC flavor, built 2025-04-04), `web-app-phpldapadmin` (`phpldapadmin/phpldapadmin:2.3.11`, built 2026-05-15) and `web-app-shopware` (`ghcr.io/shopware/docker-base:8.3`, PHP 8.3.25). `web-app-magento` keeps the 756M its `nginx.conf.sample` (`PHP_VALUE`) and `pub/.user.ini` set and keeps `post_max_size` below a quarter of that limit.
- Drop the `php_memory_unlimited` mount ([memory-unlimited.ini](roles/sys-svc-compose-php/files/memory-unlimited.ini), `memory_limit = -1`) from each role whose image ships PHP 8.2.35, 8.3.36, 8.4.27, 8.5.12 or later, and the file, its install task and `FILE_PHP_MEMORY_UNLIMITED_INI` once no role mounts it. It is the interim mitigation for CVE-2026-109618 and overrides the per-role `service_memory_mb` share, so a single request can exhaust the container instead of hitting a PHP limit. Still mounted by `web-app-magento` (`markoshust/magento-php:8.4-fpm`, PHP 8.4.21), `web-app-mailu` (`ghcr.io/heviat/webmail:2024.06`, the OIDC flavor, built 2025-04-04), `web-app-phpldapadmin` (`phpldapadmin/phpldapadmin:2.3.11`, built 2026-05-15) and `web-app-shopware` (`ghcr.io/shopware/docker-base:8.3`, PHP 8.3.25). `web-app-fusiondirectory` (`nfrastack/fusiondirectory:1.5`, PHP 7.4.33 on Alpine 3.15, end of life and never patched) carries it as `PHP_MEMORY_LIMIT=-1` in `templates/env.j2` instead (the image's pool sets `php_admin_value[memory_limit]` from it, so a conf.d ini cannot reach FPM) and drops it under the same rule. `web-app-magento` keeps the 756M its `nginx.conf.sample` (`PHP_VALUE`) and `pub/.user.ini` set and keeps `post_max_size` below a quarter of that limit.
- Fork [rpardini/docker-registry-proxy](https://github.com/rpardini/docker-registry-proxy) and teach the bundled nginx to handle GCP Artifact Registry redirect URLs (`/artifacts-downloads/namespaces/<ns>/repositories/<repo>/downloads/<token>`). Currently `gcr.io` blob pulls are bypassed via `NO_PROXY=…,gcr.io,storage.googleapis.com,googleusercontent.com` in [compose/registry-cache/proxy.conf](compose/registry-cache/proxy.conf), which is a workaround that disables caching for those registries. Long-term: rebase upstream's `proxy.conf` / `sub_filter` rules to recognise the new redirect shape, ship the fix in our fork, point `compose/registry-cache/Dockerfile` (or its image tag) at the fork, and drop the `NO_PROXY` bypass.
## Testing
+1 -1
View File
@@ -16,7 +16,7 @@ from utils.roles.custom import CustomRoleError
if TYPE_CHECKING:
from .declaration import Source
_COPY_IGNORE = shutil.ignore_patterns(".git", "__pycache__")
_COPY_IGNORE = shutil.ignore_patterns(".git", ".claude", "__pycache__")
_DROPPED_ENV = frozenset(
{
"GIT_ALTERNATE_OBJECT_DIRECTORIES",
@@ -14,9 +14,9 @@ As an operator, I want the Nextcloud Talk high-performance backend and recording
- [x] The role renders the signaling `server.conf`, `janus.jcfg` and the recording `server.conf`; no file inside an image regenerates or overwrites them at container start.
- [x] The Janus config drift guard in `tasks/01_fullstack.yml` and the `janus_config_drift` filter are removed together with their tests.
- [ ] `utils/update` reports a newer upstream version for each of the four pins, and `tests/lint/ansible/services/test_version_pins_are_watched` passes for them.
- [ ] The public signaling endpoint stays `/standalone-signaling/`, and the Talk admin settings show the pinned signaling version as "Running version" with `chat-relay` not listed as missing.
- [x] The public signaling endpoint stays `/standalone-signaling/`, and the Talk admin settings show the pinned signaling version as "Running version" with `chat-relay` not listed as missing.
- [ ] Direct media keeps working: with `services.talk.network_mode: host`, `nat_1_1_mapping` carries the public IP and `test-talk-call-capacity.js` passes.
- [ ] A recording started from a Talk call through the recording server produces a file, verified by a Playwright or CLI test.
- [x] A recording started from a Talk call through the recording server produces a file, verified by a Playwright or CLI test.
- [ ] Each new service has its own healthcheck and resource limits, and in swarm the services Janus shares a network namespace with are placed on the same node.
- [ ] Deploying over an existing `aio-talk` installation removes the old container or swarm service and starts the new services with the existing signaling, TURN and recording secrets in one deploy.
- [x] No secret of the new services appears in a container's argv; secrets reach them through config files or environment.
+1 -1
View File
@@ -33,7 +33,7 @@ deploy = [
"requests",
]
dev = [
"ruff==0.17.2",
"ruff==0.17.3",
"import-linter",
"pre-commit",
"git-maintainer-tools",
@@ -30,6 +30,8 @@ services:
{{ service_name }}:
{{ lookup('container_image', application_id, service_name) }}
{{ lookup('container_volumes', service_name) | indent(4) }}
environment:
JAVA_OPTS: "-Dpostgres.transactionIsolation=READ_COMMITTED"
{% set service_name = 'bbb-graphql-server' %}
{{ service_name }}:
{{ lookup('container_image', application_id, service_name) }}
@@ -64,11 +64,10 @@ broker returns HTTP 401, there is no anonymous fallback by design.
createSession failure and the user sees an HTML error page on the
handoff route. Re-trying the visit after PDS recovery completes
the handoff.
- **Encrypted attribute decrypt failure:** indicates a key mismatch
between the SPI and the broker. The broker rejects the request
with HTTP 500 rather than risk silently re-provisioning a fresh
PDS account that loses the user's existing posts. Recovery: rotate
the key and clear the attribute (manual op for now, key rotation
is in the doc 013 Future Hardening section).
- **Stored app-password does not decrypt:** `BLUESKY_BRIDGE_ENCRYPTION_KEY`
changed since the entry in `BROKER_STORE_PATH` was written. The
broker resets the password of the stored DID through the PDS admin
API (`PDS_ADMIN_PASSWORD`), stores it encrypted with the current key
and completes the handoff. Account, handle and posts stay.
- **social-app upstream error:** the broker forwards the upstream's
error response.
@@ -300,11 +300,24 @@ function sessionFor(entry) {
return pdsCreateSession({ handle: entry.handle, password: decrypt(entry.appPasswordEnc) });
}
async function storedSession(kcUsername, entry) {
function storedPassword(entry) {
try {
return await sessionFor(entry);
} catch (err) {
if (!pdsMessageIncludes(err, "Invalid identifier or password")) throw err;
return decrypt(entry.appPasswordEnc);
} catch {
return null;
}
}
async function storedSession(kcUsername, entry) {
const stored = storedPassword(entry);
if (stored === null) {
console.log(`[broker] stored app-password of ${logSafe(kcUsername)} does not decrypt with the current key → reset by DID`);
} else {
try {
return await pdsCreateSession({ handle: entry.handle, password: stored });
} catch (err) {
if (!pdsMessageIncludes(err, "Invalid identifier or password")) throw err;
}
}
const password = crypto.randomBytes(18).toString("base64url");
await pdsAdminUpdatePassword(entry.did, password);
+1
View File
@@ -75,6 +75,7 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (
- **Outbound mail (v1):** when `web-app-stalwart` is present, Frappe's outbound Email Account is auto-configured against the central SMTP endpoint. **IMAP inbound (mail-to-Communication) is deferred to a follow-up requirement** (no precedent in the repo for auto-provisioning role-owned Stalwart mailboxes yet).
- **Wizard bypass:** first deploy seeds the API-bot user, marks `setup_complete=1` in System Settings, fills its empty country and currency with the Company's, and skips the Frappe setup wizard entirely.
- **Company ready to invoice:** every deploy ensures the Company named `service_provider.company.titel` in the country `service_provider.company.address.country`, with the currency ERPNext's country data assigns to that country, plus a fiscal year for today, the tax accounts `VAT 19%`, `VAT 7%`, `VAT 20%`, `Reverse Charge` and `Export` under `Duties and Taxes` and a default cash account, so quotes, orders and invoices work right after the deploy. The abbreviation ERPNext appends to every account is `applications.web-app-erpnext.services.erpnext.company.abbr`, the initials of the company name unless the inventory sets it. A Company that already exists keeps its own values, and a country ERPNext does not know, such as the placeholder default, fails the deploy.
- **Server Scripts switch:** `applications.web-app-erpnext.services.erpnext.server_scripts` (default `false`) sets `server_script_enabled` in `common_site_config.json` to `1` or `0` on every deploy and restarts the Frappe processes only when the value changes.
## Quick Setup
@@ -0,0 +1,37 @@
"""Align server_script_enabled in common_site_config.json with the declared switch.
Frappe reads the key only from common_site_config.json and tests it with bool(),
so it is written as the integer 1 or 0: the string "0" that a plain
`bench set-config -g` stores would leave server scripts enabled.
Runs on the stdin of the bench Python inside the ERPNext backend container, from
the sites directory, and prints one JSON line {"changed": <bool>}.
Environment:
SERVER_SCRIPTS_ENABLED: "true" or "false" in any case, the declared switch.
"""
import json
import os
from pathlib import Path
KEY = "server_script_enabled"
CONFIG = Path("common_site_config.json")
def main():
enabled = {"true": 1, "false": 0}[os.environ["SERVER_SCRIPTS_ENABLED"].lower()]
config = json.loads(CONFIG.read_text(encoding="utf-8"))
changed = config.get(KEY) != enabled
if changed:
config[KEY] = enabled
staged = CONFIG.with_name(CONFIG.name + ".tmp")
staged.write_text(
json.dumps(config, indent=1, sort_keys=True) + "\n", encoding="utf-8"
)
staged.replace(CONFIG)
print(json.dumps({"changed": changed}))
if __name__ == "__main__":
main()
+1
View File
@@ -79,6 +79,7 @@ erpnext:
service_key: frontend
company:
abbr: "{{ service_provider.company.titel.split() | map('first') | join | upper }}"
server_scripts: false
run_after:
- web-app-keycloak
- web-app-stalwart
@@ -78,3 +78,21 @@
CURRENCY: "{{ (erpnext_company_apply.stdout_lines | last | from_json).currency }}"
register: erpnext_api_bot_apply
changed_when: false
- name: "📜 align server_script_enabled with services.erpnext.server_scripts"
ansible.builtin.shell:
cmd: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node
container exec -i
-e SERVER_SCRIPTS_ENABLED
{{ ERPNEXT_BACKEND_LOCAL_CID | quote }}
{{ ERPNEXT_BENCH_PYTHON }}
stdin: "{{ lookup('ansible.builtin.file', role_path ~ '/files/python/apply/server_scripts.py') }}"
delegate_to: "{{ ERPNEXT_BACKEND_HOST_NODE }}"
environment:
SERVER_SCRIPTS_ENABLED: "{{ ERPNEXT_SERVER_SCRIPTS_ENABLED }}"
register: erpnext_server_scripts_apply
changed_when: (erpnext_server_scripts_apply.stdout_lines | last | from_json).changed
- name: "🔄 Restart Frappe so every worker reads the new server_script_enabled"
ansible.builtin.include_tasks: 02_bench_bootstrap/restart_frappe.yml
when: erpnext_server_scripts_apply is changed
@@ -28,51 +28,5 @@
changed_when: true
timeout: "{{ lookup('timeout', 1800) | int }}"
- name: "🔄 restart Frappe containers to pick up newly-installed app (compose mode)"
ansible.builtin.shell: |
set -euo pipefail
container restart \
{{ ERPNEXT_BACKEND_CONTAINER }} \
{{ ERPNEXT_FRONTEND_CONTAINER }} \
{{ ERPNEXT_WEBSOCKET_CONTAINER }} \
{{ ERPNEXT_SCHEDULER_CONTAINER }} \
{{ ERPNEXT_QUEUE_SHORT_CONTAINER }} \
{{ ERPNEXT_QUEUE_LONG_CONTAINER }}
changed_when: false
when: DEPLOYMENT_MODE == 'compose'
- name: "🔄 restart Frappe services to pick up newly-installed app (swarm mode)"
ansible.builtin.command: >-
container service update --force {{ entity_name }}_{{ item }}
loop:
- backend
- frontend
- websocket
- scheduler
- queue-short
- queue-long
changed_when: false
timeout: "{{ lookup('timeout', 600) | int }}"
when: DEPLOYMENT_MODE == 'swarm'
- name: "🔍 Re-resolve ERPNext backend node + CID after restart"
ansible.builtin.include_tasks: "{{ lookup('path_absolute', 'roles/sys-svc-compose/tasks/utils/swarm/resolve_host_cid.yml') }}"
vars:
resolve_service_key: backend
resolve_output_prefix: ERPNEXT_BACKEND
resolve_compose_cid: "{{ ERPNEXT_BACKEND_CONTAINER }}"
resolve_converge_retries: 60
resolve_converge_delay: 5
- name: "⏳ wait for {{ ERPNEXT_BACKEND_CONTAINER }} gunicorn TCP port after restart"
ansible.builtin.shell: |
set -eo pipefail
# nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node
container exec {{ ERPNEXT_BACKEND_LOCAL_CID | quote }} \
bash -c 'exec 3<>/dev/tcp/localhost/{{ ERPNEXT_BACKEND_PORT }} && exec 3<&- 3>&-'
delegate_to: "{{ ERPNEXT_BACKEND_HOST_NODE }}"
register: erpnext_backend_ready_post_restart
changed_when: false
retries: 60
delay: 5
until: erpnext_backend_ready_post_restart.rc == 0
- name: "🔄 Restart Frappe to pick up the newly-installed app"
ansible.builtin.include_tasks: restart_frappe.yml
@@ -0,0 +1,49 @@
---
- name: "🔄 restart Frappe containers (compose mode)"
ansible.builtin.shell: |
set -euo pipefail
container restart \
{{ ERPNEXT_BACKEND_CONTAINER }} \
{{ ERPNEXT_FRONTEND_CONTAINER }} \
{{ ERPNEXT_WEBSOCKET_CONTAINER }} \
{{ ERPNEXT_SCHEDULER_CONTAINER }} \
{{ ERPNEXT_QUEUE_SHORT_CONTAINER }} \
{{ ERPNEXT_QUEUE_LONG_CONTAINER }}
changed_when: false
when: DEPLOYMENT_MODE == 'compose'
- name: "🔄 restart Frappe services (swarm mode)"
ansible.builtin.command: >-
container service update --force {{ entity_name }}_{{ item }}
loop:
- backend
- frontend
- websocket
- scheduler
- queue-short
- queue-long
changed_when: false
timeout: "{{ lookup('timeout', 600) | int }}"
when: DEPLOYMENT_MODE == 'swarm'
- name: "🔍 Re-resolve ERPNext backend node + CID after restart"
ansible.builtin.include_tasks: "{{ lookup('path_absolute', 'roles/sys-svc-compose/tasks/utils/swarm/resolve_host_cid.yml') }}"
vars:
resolve_service_key: backend
resolve_output_prefix: ERPNEXT_BACKEND
resolve_compose_cid: "{{ ERPNEXT_BACKEND_CONTAINER }}"
resolve_converge_retries: 60
resolve_converge_delay: 5
- name: "⏳ wait for {{ ERPNEXT_BACKEND_CONTAINER }} gunicorn TCP port after restart"
ansible.builtin.shell: |
set -eo pipefail
# nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node
container exec {{ ERPNEXT_BACKEND_LOCAL_CID | quote }} \
bash -c 'exec 3<>/dev/tcp/localhost/{{ ERPNEXT_BACKEND_PORT }} && exec 3<&- 3>&-'
delegate_to: "{{ ERPNEXT_BACKEND_HOST_NODE }}"
register: erpnext_backend_ready_post_restart
changed_when: false
retries: 60
delay: 5
until: erpnext_backend_ready_post_restart.rc == 0
+2
View File
@@ -29,6 +29,8 @@ ERPNEXT_COMPANY_NAME: "{{ service_provider.company.titel }}"
ERPNEXT_COMPANY_ABBR: "{{ lookup('config', application_id, 'services.erpnext.company.abbr') }}"
ERPNEXT_COMPANY_COUNTRY: "{{ service_provider.company.address.country }}"
ERPNEXT_SERVER_SCRIPTS_ENABLED: "{{ lookup('config', application_id, 'services.erpnext.server_scripts') | bool }}"
ERPNEXT_OIDC_ENABLED: "{{ lookup('config', application_id, 'services.sso.enabled') | bool }}"
ERPNEXT_LDAP_ENABLED: "{{ lookup('config', application_id, 'services.ldap.enabled') | bool }}"
ERPNEXT_EMAIL_ENABLED: "{{ lookup('config', application_id, 'services.email.enabled') | bool }}"
+1 -1
View File
@@ -1,4 +1,4 @@
# To-dos
- Move off PHP 7.4. The pinned `nfrastack/fusiondirectory:1.5` image builds on `ghcr.io/nfrastack/container-nginx-php-fpm` with PHP 7.4 on Alpine 3.15, which is end of life and gets no fix for CVE-2026-109618 and CVE-2026-109619.
- No FusionDirectory release runs on a supported PHP yet: upstream certifies 1.4 to 1.6 for PHP 7.4 only, `nfrastack/container-fusiondirectory` builds its image only on PHP 7.4 and Alpine 3.15, and the PHP 8 port lives on upstream's unreleased `php8` branch. Until then the shared hardening ini mounted at `/etc/php7/conf.d/zz-infinito-hardening.ini` carries the mitigations for both CVEs. Options: rebuild the image on upstream's first `php8` release, or build `nfrastack/container-fusiondirectory` against a pinned `php8` commit on `container-nginx-php-fpm` 8.4 before that release.
- No FusionDirectory release runs on a supported PHP yet: upstream certifies 1.4 to 1.6 for PHP 7.4 only, `nfrastack/container-fusiondirectory` builds its image only on PHP 7.4 and Alpine 3.15, and the PHP 8 port lives on upstream's unreleased `php8` branch. Until then the shared hardening ini mounted at `/etc/php7/conf.d/zz-infinito-hardening.ini` carries the CVE-2026-109619 mitigation and `PHP_MEMORY_LIMIT=-1` in `templates/env.j2` the CVE-2026-109618 one. Options: rebuild the image on upstream's first `php8` release, or build `nfrastack/container-fusiondirectory` against a pinned `php8` commit on `container-nginx-php-fpm` 8.4 before that release.
@@ -8,3 +8,4 @@ LDAP01_PORT={{ lookup('config', 'svc-db-openldap', 'services.openldap.ports.loca
LDAP01_ADMIN_DN={{ LDAP.DN.ADMINISTRATOR.DATA }}
LDAP01_ADMIN_PASS={{ LDAP.BIND_CREDENTIAL | dotenv_quote }}
LDAP01_BASE_DN={{ LDAP.DN.ROOT }}
PHP_MEMORY_LIMIT=-1
+1
View File
@@ -87,6 +87,7 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (
- **Customizable Webmail and Administration:** Provides a modern web interface for managing emails and administrative tasks with ease.
- **Flexible Deployment:** Easily scale and customize using Docker Compose, with configurable settings for networking, storage, and external services.
- **OIDC Support:** Optionally integrate with OpenID Connect for centralized authentication across your services.
- **DMARC Reports:** The published `_dmarc` record sends aggregate reports (`rua`) to `<username>@<mail domain>` of the user key in `services.mailu.dmarc.rua_user` (`administrator` unless the inventory names another one). That user MUST carry a `mailbox` account or a `forward` to a user that carries one and whose `email` is that mailbox (`<username>@<mail domain>`), otherwise the deploy fails before any record is written.
## Quick Setup
+2
View File
@@ -46,6 +46,8 @@ mailu:
project_hard_restart: true
min_storage: 20GB
domain: "{{ lookup('email').domain }}"
dmarc:
rua_user: administrator
ports:
local:
http: 8007
+20 -1
View File
@@ -1,4 +1,23 @@
---
- name: "✅ Assert the DMARC aggregate report address reaches a Mailu mailbox or alias"
ansible.builtin.assert:
that:
- >-
'mailbox' in lookup('users', MAILU_DMARC_RUA_USER).accounts
or (
lookup('users', MAILU_DMARC_RUA_USER).forward | length > 0
and 'mailbox' in lookup('users', lookup('users', MAILU_DMARC_RUA_USER).forward).accounts
and lookup('users', lookup('users', MAILU_DMARC_RUA_USER).forward).email
== lookup('users', lookup('users', MAILU_DMARC_RUA_USER).forward).username ~ '@' ~ MAILU_DOMAIN
)
fail_msg: >-
services.mailu.dmarc.rua_user '{{ MAILU_DMARC_RUA_USER }}' carries
neither a mailbox account nor a forward to a user with one whose email
is that mailbox (<username>@{{ MAILU_DOMAIN }}), so {{ MAILU_DMARC_RUA }}
reaches no Mailu mailbox and DMARC aggregate reports bounce or leave
the server.
quiet: true
- name: "🌐 DNS (Cloudflare) for Mailu"
include_role:
name: sys-dns-cloudflare-records
@@ -23,7 +42,7 @@
zone: "{{ MAILU_DOMAIN_DNS_ZONE }}"
name: "_dmarc.{{ MAILU_DOMAIN_DNS_ZONE }}"
solo: true
value: '"v=DMARC1; p=reject; ruf=mailto:{{ MAILU_DMARC_RUF }}; adkim=s; aspf=s"'
value: '"v=DMARC1; p=reject; rua=mailto:{{ MAILU_DMARC_RUA }}; ruf=mailto:{{ MAILU_DMARC_RUF }}; adkim=s; aspf=s"'
- type: TXT
zone: "{{ MAILU_DOMAIN_DNS_ZONE }}"
name: "dkim._domainkey.{{ MAILU_DOMAIN_DNS_ZONE }}"
+1 -1
View File
@@ -20,7 +20,7 @@ RELAYNETS=
RELAYHOST=
FETCHMAIL_DELAY=600
RECIPIENT_DELIMITER=+
DMARC_RUA=admin
DMARC_RUA={{ MAILU_DMARC_RUA_LOCALPART }}
DMARC_RUF=admin
WELCOME=false
WELCOME_SUBJECT=Welcome to your new email account
+3
View File
@@ -48,6 +48,9 @@ MAILU_EXTRA_HOSTS: "{{ query('mailu_extra_hosts') }}"
MAILU_USERS: "{{ lookup('mailu_users') }}"
MAILU_DOCKER_FLAVOR: "{{ 'ghcr.io/heviat' if MAILU_OIDC_ENABLED | bool else 'ghcr.io/mailu' }}"
MAILU_DMARC_RUF: "{{ lookup('config', application_id, 'users.administrator.email') }}"
MAILU_DMARC_RUA_USER: "{{ lookup('config', application_id, 'services.mailu.dmarc.rua_user') }}"
MAILU_DMARC_RUA_LOCALPART: "{{ lookup('users', MAILU_DMARC_RUA_USER).username }}"
MAILU_DMARC_RUA: "{{ MAILU_DMARC_RUA_LOCALPART }}@{{ MAILU_DOMAIN }}"
MAILU_DKIM_KEY_FILE: "{{ MAILU_DOMAIN }}.dkim.key"
MAILU_DKIM_KEY_PATH: "/dkim/{{ MAILU_DKIM_KEY_FILE }}"
MAILU_NGINX_HOST_DIR: "{{ [ MAILU_OVERRIDES_DIR, 'nginx' ] | path_join }}"
+2
View File
@@ -50,6 +50,8 @@
container_backup:
enabled: false
shared: false # nocheck: variants-static-override backup shared stays literally true
element:
force_verification: false
matrix:
flavor: ansible
coturn:
@@ -0,0 +1,39 @@
<?php
// nocheck: mirrored-unit-test - needs a booted Nextcloud, which only exists inside the container
// Params: $argv[1] the Nextcloud server root; $occBatch, assigned in front of this script on
// stdin, the base64 of a JSON list of {"argv": [occ arguments], "label": "<call name>"}.
// Output: what occ prints for each call; on the first call that exits non-zero or throws,
// "occ <command> failed for <label>: <reason>" (or "occ bootstrap failed: <reason>") on stderr
// and exit code 1.
define("OC_CONSOLE", 1);
[, $serverRoot] = $argv;
$calls = json_decode(base64_decode($occBatch, true), true, 512, JSON_THROW_ON_ERROR);
$failure = "occ bootstrap failed";
try {
require_once $serverRoot . "/lib/base.php";
chdir($serverRoot);
$application = \OCP\Server::get(\OC\Console\Application::class);
$application->loadCommands(
new \Symfony\Component\Console\Input\ArgvInput(["occ", ...$calls[0]["argv"]]),
new \Symfony\Component\Console\Output\ConsoleOutput(),
);
$application->setAutoExit(false);
foreach ($calls as $call) {
$failure = "occ " . $call["argv"][0] . " failed for " . $call["label"];
$rc = $application->run(new \Symfony\Component\Console\Input\ArgvInput(["occ", ...$call["argv"]]));
if ($rc !== 0) {
throw new \RuntimeException("exit code " . $rc);
}
}
} catch (\Throwable $e) {
fwrite(STDERR, $failure . ": " . $e->getMessage() . "\n");
exit(1);
}
@@ -4,8 +4,10 @@ join an `occ app:install` that runs in the background.
"""
import json
import re
_INCOMPAT_MARKER = "is not compatible with this version of the server"
_ENABLED_LINE = re.compile(r"^\S+ enabled$", re.MULTILINE)
_ALREADY_MARKER = "already installed"
_UNAVAILABLE_MARKER = "was not found on the appstore"
_TRANSPORT_MARKERS = ("cURL error", "ConnectException")
@@ -31,6 +33,8 @@ def nextcloud_install_status(install_result):
(drives `until` and inverts to `failed_when`)
- changed : plugin was actually installed in this run
- runnable : plugin is available for enable/configure downstream
- enabled : the install enabled the plugin too; its stdout holds
an ``<app> enabled`` line
"""
if not isinstance(install_result, dict):
raise TypeError(
@@ -55,6 +59,7 @@ def nextcloud_install_status(install_result):
"ok": success or already or incompatible,
"changed": success and not already,
"runnable": success or already,
"enabled": success and _ENABLED_LINE.search(stdout) is not None,
}
@@ -78,8 +78,44 @@ def nextcloud_plugin_config_pending(specs, app_config):
return pending
def nextcloud_occ_config_set_batch(pending):
"""The ``occ config:app:set`` calls that write the pending entries.
Args:
pending: ``nextcloud_plugin_config_pending`` entries.
Returns:
list of dicts in entry order, the calls files/php/utils/occ_batch.php runs:
argv - the ``occ`` arguments; a typed entry adds
``--no-interaction --type=<type>``
label - ``appid/configkey``
"""
batch = []
for item in pending:
typed = (
["--no-interaction", "--type=" + item["type"]]
if item["type"] is not None
else []
)
batch.append(
{
"argv": [
"config:app:set",
item["appid"],
item["configkey"],
*typed,
"--value",
item["value"],
],
"label": item["appid"] + "/" + item["configkey"],
}
)
return batch
class FilterModule:
def filters(self):
return {
"nextcloud_plugin_config_pending": nextcloud_plugin_config_pending,
"nextcloud_occ_config_set_batch": nextcloud_occ_config_set_batch,
}
@@ -43,6 +43,9 @@ config:
- appid: "spreed"
configkey: "internal_secret"
configvalue: "{{ NEXTCLOUD_HPB_INTERNAL_SECRET }}"
- appid: "spreed"
configkey: "call_recording_summary"
configvalue: "{{ 'yes' if (lookup('config', 'web-app-nextcloud', 'addons.integration_openai.enabled') | bool) else 'no' }}"
- appid: "spreed"
configkey: "recording_servers"
type: array
@@ -81,4 +81,11 @@
or (install_result | nextcloud_install_status).unavailable
- include_tasks: 03_enable.yml
when: (install_result | nextcloud_install_status).runnable
when:
- (install_result | nextcloud_install_status).runnable
- not (install_result | nextcloud_install_status).enabled
- name: "🧮 Queue {{ plugin_key }} for the batched configuration"
ansible.builtin.set_fact:
nextcloud_plugin_configure_keys: "{{ nextcloud_plugin_configure_keys + [plugin_key] }}"
when: (install_result | nextcloud_install_status).enabled
@@ -22,14 +22,12 @@
| nextcloud_plugin_config_pending(nextcloud_app_config.stdout | from_json) }}
no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
- name: "🔧 Set the pending plugin configuration (one exec, occ calls on stdin)"
- name: "🔧 Set the pending plugin configuration (one occ boot, values on stdin)"
ansible.builtin.shell:
cmd: "{{ NEXTCLOUD_DOCKER_EXEC_INTERACTIVE }} sh -s"
stdin: |
set -eu
{% for item in nextcloud_plugin_config_pending %}
{{ NEXTCLOUD_INTERNAL_OCC_COMMAND | quote }} config:app:set {{ item.appid | quote }} {{ item.configkey | quote }}{{ (' --no-interaction --type=' ~ item.type) if item.type is not none else '' }} --value {{ item.value | quote }} || { echo {{ ('config:app:set failed for ' ~ item.appid ~ '/' ~ item.configkey) | quote }} >&2; exit 1; }
{% endfor %}
cmd: "{{ NEXTCLOUD_DOCKER_EXEC_INTERACTIVE }} php -- {{ NEXTCLOUD_DOCKER_WORK_DIRECTORY | quote }}"
stdin: >-
<?php $occBatch = '{{ nextcloud_plugin_config_pending | nextcloud_occ_config_set_batch | to_json | b64encode }}'; ?>{{ lookup('file', 'php/utils/occ_batch.php')
}}
when: nextcloud_plugin_config_pending | length > 0
register: nextcloud_plugin_config_set
changed_when: nextcloud_plugin_config_set.stdout is search(' set to ')
@@ -48,9 +48,9 @@ Each `tasks/02_manager_ops/06_plugin/<step>.yml` file owns exactly one OCC stage
| File | Stage | OCC command | Continues to next stage when |
|---|---|---|---|
| [02_install.yml](./02_install.yml) | install | `occ app:install` | install result is `runnable` (success or already installed) |
| [02_install.yml](./02_install.yml) | install | `occ app:install` | install result is `runnable` (success or already installed); an install that printed `<app> enabled` queues the plugin for the configure stage directly and skips the enable stage |
| [03_enable.yml](./03_enable.yml) | enable | `occ app:enable` | enable result is not flagged `incompatible`; the plugin is then queued for the configure stage |
| [04_configure.yml](./04_configure.yml) | configure | one `occ config:list`, then `occ config:app:set` for the values that differ, plus optional hook | (terminal stage) |
| [04_configure.yml](./04_configure.yml) | configure | one `occ config:list`, then `config:app:set` for the values that differ in one occ boot, plus optional hook | (terminal stage) |
## Pinned archives
@@ -68,7 +68,7 @@ An addon MAY pin a release through `config.archive`. Before `occ app:install`, [
1. It loads the addon spec of every queued plugin.
2. It reads `occ config:list` once and keeps an entry only when Nextcloud does not already hold the exact value `occ config:app:set` would store. Entries with a `type`, sensitive values that `config:list` masks, booleans, floats and structured values are always kept.
3. It writes the kept entries in spec order through one shell; the first failing `config:app:set` stops the shell and names its `appid/configkey` on stderr. The task reports changed only when `occ` answers `set to`.
3. It writes the kept entries in spec order through one php process: [files/php/utils/occ_batch.php](../../../files/php/utils/occ_batch.php) boots the occ console the way Nextcloud's `console.php` does and runs one `config:app:set` per entry, with the arguments the filter `nextcloud_occ_config_set_batch` builds. The calls arrive as base64 JSON on stdin, in front of the script. The first failing call stops the process and names its `appid/configkey` on stderr. The task reports changed only when `occ` answers `set to`.
4. It runs the hook of every queued plugin in queue order.
The keep or drop decision lives in the filter `nextcloud_plugin_config_pending` ([filter_plugins/nextcloud_plugin_config.py](../../../filter_plugins/nextcloud_plugin_config.py)).
-1
View File
@@ -27,7 +27,6 @@ NEXTCLOUD_ADMIN_PASSWORD={{ NEXTCLOUD_ADMINISTRATOR_PASSWORD | dotenv_quote }}
NEXTCLOUD_INIT_GATE_USER={{ NEXTCLOUD_DOCKER_USER }}
NEXTCLOUD_TRUSTED_DOMAINS={{ NEXTCLOUD_DOMAIN }}
TRUSTED_PROXIES={{ NEXTCLOUD_TRUSTED_PROXIES | join(' ') }}
OVERWRITECLIURL={{ NEXTCLOUD_URL }}
OVERWRITEPROTOCOL={{ NEXTCLOUD_PROTOCOL }}
+1 -1
View File
@@ -53,7 +53,7 @@ pretix:
swarm:
enabled: true
image: pretix/standalone
version: 2026.8.1
version: 2026.9.0
custom: true
name: pretix
min_storage: 1003MB
+1 -1
View File
@@ -47,7 +47,7 @@ shopware:
swarm:
enabled: true
image: composer
version: "2.10"
version: "2.11"
custom: true
min_storage: 278MB
package:
+6 -1
View File
@@ -224,6 +224,11 @@ account (or the Keycloak SSO token when SSO is enabled).
per-volume stop/start cycles.
- **ClamAV runs fail-open with short milter timeouts** so mail keeps flowing
while clamd warms its signature database instead of blocking the DATA stage.
- **DMARC aggregate reports go to the mailbox of
`services.stalwart.dmarc.rua_user`** (the `administrator` user unless the
inventory names another user key). The `_dmarc` record publishes
`rua=mailto:<username>@<mail domain>`, and a deploy whose report user carries
no `mailbox` account fails before any record is written.
- **`config.json` is bootstrap-only** (data store selection); all other
configuration lives in the database and is provisioned over JMAP
(`templates/jmap/*.json.j2`).
@@ -472,7 +477,7 @@ Set `INFINITO_MAIL_PRODUCTION_HOST` to the server's public FQDN (in CI, a GitHub
| A / AAAA | the mail host resolves |
| MX | the zone's only MX is the mail host (the role publishes it `solo`) |
| SPF | exactly one `v=spf1` record, authorising `mx` or `a:<mailhost>` |
| DMARC | `_dmarc` publishes a `v=DMARC1` record with a policy |
| DMARC | `_dmarc` publishes a `v=DMARC1` record with a policy whose `rua` names the deployed aggregate report mailbox |
| DKIM | the selector this deploy read back publishes a non-empty `v=DKIM1` key |
| SRV | `_submissions`/465, `_imaps`/993, `_pop3s`/995 point at the mail host |
| PTR | rDNS does not contradict the mail host (absent → warning; rDNS is provisioned for Hetzner only) |
@@ -23,6 +23,7 @@ import struct
import sys
import time
from pathlib import Path
from urllib.parse import unquote
TYPE_A = 1
TYPE_CNAME = 5
@@ -313,17 +314,62 @@ def check_spf(args, resolvers, report: Report) -> None:
report.ok(f"SPF of {args.mail_domain}: {value}")
def dmarc_tags(record: str) -> dict[str, str]:
"""Split a DMARC record into its ``tag=value`` pairs, tags lowercased.
Raises:
ValueError: a tag occurs twice, which invalidates the whole record
(RFC 6376 3.2 tag-list, adopted by RFC 7489 6.3).
"""
tags: dict[str, str] = {}
for field in record.split(";"):
tag, separator, value = field.partition("=")
if not separator:
continue
tag = tag.strip().lower()
if tag in tags:
raise ValueError(f"duplicate tag {tag}=")
tags[tag] = value.strip()
return tags
def dmarc_report_addresses(uris: str) -> list[str]:
"""The lowercased ``mailto:`` addresses of a ``rua``/``ruf`` value.
A URI may end in a ``!<size>`` report size limit (RFC 7489 6.2), which is
not part of the address. The address itself may be percent-encoded and
followed by ``?`` header fields (RFC 6068 2).
"""
addresses: list[str] = []
for uri in uris.split(","):
scheme, separator, target = uri.strip().split("!", 1)[0].partition(":")
if separator and scheme.lower() == "mailto":
addresses.append(unquote(target.split("?", 1)[0]).lower())
return addresses
def check_dmarc(args, resolvers, report: Report) -> None:
"""_dmarc TXT exists, is a DMARC1 record and declares a policy."""
"""_dmarc TXT is a DMARC1 record with a policy and the deployed rua address."""
name = f"_dmarc.{args.mail_domain}"
records = [str(r) for r in query(name, TYPE_TXT, resolvers, args.timeout)]
dmarc = [r for r in records if r.lower().startswith("v=dmarc1")]
if not dmarc:
report.fail(f"{name} has no v=DMARC1 TXT record")
return
if "p=" not in dmarc[0].lower():
try:
tags = dmarc_tags(dmarc[0])
except ValueError as error:
report.fail(f"{name} is not a valid DMARC record, {error}: {dmarc[0]}")
return
if not tags.get("p"):
report.fail(f"{name} declares no policy (p=): {dmarc[0]}")
return
expected = args.dmarc_rua.strip().lower()
if expected not in dmarc_report_addresses(tags.get("rua", "")):
report.fail(
f"{name} sends no aggregate reports to {expected} (rua=): {dmarc[0]}"
)
return
report.ok(f"DMARC of {args.mail_domain}: {dmarc[0]}")
@@ -489,6 +535,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.add_argument(
"--mail-domain", required=True, help="Zone carrying MX/SPF/DMARC"
)
parser.add_argument(
"--dmarc-rua",
required=True,
help="Mailbox the DMARC record must name as aggregate report address (rua)",
)
parser.add_argument(
"--dkim-selector", default="", help="Selector published by the server"
)
+3 -1
View File
@@ -44,7 +44,7 @@ stalwart:
placement: manager
image: stalwartlabs/stalwart
version: "v1.0.1"
webui_version: "v2.0.2"
webui_version: "v2.0.3"
update:
key: webui_version
type: git_tags
@@ -64,6 +64,8 @@ stalwart:
# nocheck: min-storage-headroom Reason: the mail store (maildir blobs, FTS index, queue) lives in this service's volumes and dwarfs the image transfer size
min_storage: 20GB
domain: "{{ lookup('email').domain }}"
dmarc:
rua_user: administrator
ports:
local:
http: 8095
@@ -1,4 +1,14 @@
---
- name: "✅ Assert the DMARC aggregate report address reaches a Stalwart mailbox"
ansible.builtin.assert:
that:
- "'mailbox' in lookup('users', STALWART_DMARC_RUA_USER).accounts"
fail_msg: >-
services.stalwart.dmarc.rua_user '{{ STALWART_DMARC_RUA_USER }}' carries
no mailbox account, so Stalwart provisions no recipient for
{{ STALWART_DMARC_RUA }} and every DMARC aggregate report bounces.
quiet: true
- name: "🔧 DNS (Cloudflare) for Stalwart"
include_role:
name: sys-dns-cloudflare-records
@@ -32,7 +42,7 @@
zone: "{{ STALWART_DOMAIN_DNS_ZONE }}"
name: "_dmarc.{{ STALWART_DOMAIN_DNS_ZONE }}"
solo: true
value: '"v=DMARC1; p=reject; ruf=mailto:{{ STALWART_DMARC_RUF }}; adkim=s; aspf=s"'
value: '"v=DMARC1; p=reject; rua=mailto:{{ STALWART_DMARC_RUA }}; ruf=mailto:{{ STALWART_DMARC_RUF }}; adkim=s; aspf=s"'
- type: TXT
zone: "{{ STALWART_DOMAIN_DNS_ZONE }}"
name: "{{ stalwart_dkim_selector }}._domainkey.{{ STALWART_DOMAIN_DNS_ZONE }}"
@@ -9,6 +9,7 @@
{{ [ role_path, 'files/python/production_facts.py' ] | path_join }}
--mail-host {{ STALWART_PRODUCTION_HOST | quote }}
--mail-domain {{ STALWART_DOMAIN | quote }}
--dmarc-rua {{ STALWART_DMARC_RUA | quote }}
{{ ('--dkim-selector ' ~ (stalwart_dkim_selector | quote))
if stalwart_dkim_selector is defined else '' }}
{{ ('--resolver ' ~ (STALWART_PRODUCTION_RESOLVER | quote))
+2
View File
@@ -91,6 +91,8 @@ STALWART_ADMIN_PASSWORD: "{{ lookup('users', 'stalwartadmin').password }}"
STALWART_USERS: "{{ lookup('stalwart_users') }}"
STALWART_DMARC_RUF: "{{ lookup('config', application_id, 'users.administrator.email') }}"
STALWART_DMARC_RUA_USER: "{{ lookup('config', application_id, 'services.stalwart.dmarc.rua_user') }}"
STALWART_DMARC_RUA: "{{ lookup('users', STALWART_DMARC_RUA_USER).username }}@{{ STALWART_DOMAIN }}"
STALWART_BOT_APP_PASSWORD_DESCRIPTION: "Infinito.Nexus submission"
STALWART_BOT_APP_PASSWORD_RETIRED_DESCRIPTION: "Infinito.Nexus submission, retired"
@@ -11,6 +11,8 @@ const SERVER = path.join(PROJECT_ROOT, "roles/web-app-bluesky/files/login-broker
const DOMAIN = "example.test";
const PUBLIC_HOST = "web.bluesky.example.test";
const ADMIN_PASSWORD = "stub-admin-password";
const KEY = Buffer.alloc(32, 7);
const ROTATED_KEY = Buffer.alloc(32, 9);
function startPds() {
const accounts = new Map();
@@ -79,7 +81,7 @@ function freePort() {
});
}
async function startBroker(pdsPort, storePath) {
async function startBroker(pdsPort, storePath, key) {
const port = await freePort();
const child = spawn(process.execPath, [SERVER], {
env: {
@@ -90,7 +92,7 @@ async function startBroker(pdsPort, storePath) {
PDS_INTERNAL_URL: `http://127.0.0.1:${pdsPort}`,
PDS_HANDLE_DOMAIN: DOMAIN,
PDS_ADMIN_PASSWORD: ADMIN_PASSWORD,
BLUESKY_BRIDGE_ENCRYPTION_KEY: `base64:${Buffer.alloc(32, 7).toString("base64")}`,
BLUESKY_BRIDGE_ENCRYPTION_KEY: `base64:${key.toString("base64")}`,
BROKER_STORE_PATH: storePath,
},
stdio: ["ignore", "pipe", "pipe"],
@@ -141,7 +143,7 @@ let broker;
test.before(async () => {
pds = await startPds();
storePath = path.join(fs.mkdtempSync(path.join(os.tmpdir(), "broker-server-")), "accounts.json");
broker = await startBroker(pds.port, storePath);
broker = await startBroker(pds.port, storePath, KEY);
});
test.after(async () => {
@@ -238,7 +240,7 @@ test("concurrent first visits create one account", async () => {
test("a broker restart keeps every account", async () => {
await stopBroker(broker);
broker = await startBroker(pds.port, storePath);
broker = await startBroker(pds.port, storePath, KEY);
const before = pds.calls.createAccount;
for (const [user, email] of [["biber", "biber@example.test"], ["kevinveenbirkenbach", "kevin@example.test"]]) {
const res = await request(broker, { user, email });
@@ -264,3 +266,18 @@ test("a password the PDS no longer accepts is reset in place", async () => {
assert.equal(pds.calls.createAccount, before);
assert.ok(pds.calls.adminUpdatePassword.includes("did:plc:i8-administrator"));
});
test("a rotated encryption key resets the stored password in place and re-encrypts it", async () => {
await stopBroker(broker);
broker = await startBroker(pds.port, storePath, ROTATED_KEY);
const accounts = pds.calls.createAccount;
const resets = pds.calls.adminUpdatePassword.length;
const res = await request(broker, { user: "mapache", email: "mapache@example.test" });
assert.equal(res.status, 200);
assert.match(res.body, /BSKY_STORAGE/);
assert.equal(pds.calls.createAccount, accounts);
assert.deepEqual(pds.calls.adminUpdatePassword.slice(resets), ["did:plc:mapache"]);
const again = await request(broker, { user: "mapache", email: "mapache@example.test" });
assert.equal(again.status, 200);
assert.equal(pds.calls.adminUpdatePassword.length, resets + 1);
});
@@ -127,6 +127,15 @@ class TestStage(_SourceFixture, unittest.TestCase):
self.assertFalse((target / "__pycache__").exists())
self.assertIn("(working tree)", origin)
def test_local_directory_skips_unreadable_agent_harness_state(self):
harness = self.repo / ROLE_PATH / ".claude" / ".cc-writes"
harness.mkdir(parents=True)
harness.chmod(0o000)
self.addCleanup(harness.chmod, 0o700)
target = self.root / "staged"
stage(ROLE, Source(str(self.repo), None, ROLE_PATH), target)
self.assertFalse((target / ".claude").exists())
def test_path_that_is_a_file_is_refused(self):
with self.assertRaisesRegex(CustomRoleError, "source.path is not a directory"):
stage(ROLE, Source(str(self.repo), "v1.0.0", "README.md"), self.root / "s")
@@ -0,0 +1,97 @@
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from utils.cache.files import read_text
from . import PROJECT_ROOT
SCRIPT = PROJECT_ROOT / "roles/web-app-erpnext/files/python/apply/server_scripts.py"
class TestServerScripts(unittest.TestCase):
def setUp(self) -> None:
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.sites = Path(tmp.name)
self.config = self.sites / "common_site_config.json"
def _seed(self, **config) -> None:
self.config.write_text(json.dumps({"redis_cache": "redis://cache", **config}))
def _run(self, enabled: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, "-I", "-"],
input=read_text(str(SCRIPT)),
cwd=self.sites,
env={**os.environ, "SERVER_SCRIPTS_ENABLED": enabled},
capture_output=True,
text=True,
check=False,
)
def _stored(self) -> dict:
text = self.config.read_text() # nocheck: cache-read tempdir fixture
return json.loads(text)
def test_enables_as_the_integer_one_and_keeps_other_keys(self) -> None:
self._seed()
result = self._run("true")
self.assertEqual(json.loads(result.stdout), {"changed": True})
self.assertEqual(self._stored()["server_script_enabled"], 1)
self.assertEqual(self._stored()["redis_cache"], "redis://cache")
def test_disables_a_value_bench_stored_as_a_string(self) -> None:
self._seed(server_script_enabled="1")
result = self._run("false")
self.assertEqual(json.loads(result.stdout), {"changed": True})
self.assertIs(type(self._stored()["server_script_enabled"]), int)
self.assertEqual(self._stored()["server_script_enabled"], 0)
def test_rewrites_the_string_zero_that_frappe_reads_as_enabled(self) -> None:
self._seed(server_script_enabled="0")
result = self._run("false")
self.assertEqual(json.loads(result.stdout), {"changed": True})
self.assertEqual(self._stored()["server_script_enabled"], 0)
def test_an_aligned_value_is_left_untouched(self) -> None:
self._seed(server_script_enabled=1)
before = self.config.read_bytes() # nocheck: cache-read tempdir fixture
result = self._run("true")
self.assertEqual(json.loads(result.stdout), {"changed": False})
after = self.config.read_bytes() # nocheck: cache-read tempdir fixture
self.assertEqual(after, before)
def test_accepts_the_capitalised_bool_ansible_renders(self) -> None:
self._seed(server_script_enabled=0)
result = self._run("True")
self.assertEqual(json.loads(result.stdout), {"changed": True})
self.assertEqual(self._stored()["server_script_enabled"], 1)
def test_an_undeclared_switch_value_fails(self) -> None:
self._seed()
result = self._run("yes")
self.assertNotEqual(result.returncode, 0)
self.assertNotIn("server_script_enabled", self._stored())
if __name__ == "__main__":
unittest.main()
@@ -85,6 +85,26 @@ class TestNextcloudInstallStatus(unittest.TestCase):
self.assertTrue(status["changed"])
self.assertTrue(status["runnable"])
def test_an_install_that_printed_its_enable_is_enabled(self):
status = _status(rc=0, stdout="deck 1.18.6 installed\ndeck enabled")
self.assertTrue(status["enabled"])
def test_an_install_without_the_enable_line_is_not_enabled(self):
self.assertFalse(_status(rc=0, stdout="deck 1.18.6 installed")["enabled"])
def test_an_already_installed_plugin_is_not_enabled_by_the_install(self):
self.assertFalse(_status(stdout="deck already installed")["enabled"])
def test_a_failed_install_is_not_enabled_whatever_it_printed(self):
status = _status(rc=1, stdout="deck 1.18.6 installed\ndeck enabled")
self.assertFalse(status["enabled"])
def test_an_enable_line_inside_an_error_is_not_enabled(self):
status = _status(
rc=0, stdout="deck 1.18.6 installed\nError: deck enabled twice"
)
self.assertFalse(status["enabled"])
def test_a_non_dict_result_is_rejected(self):
with self.assertRaises(TypeError):
MODULE.nextcloud_install_status("rc=0")
@@ -25,6 +25,7 @@ config_mod = _load_module(
)
nextcloud_plugin_config_pending = config_mod.nextcloud_plugin_config_pending
nextcloud_occ_value = config_mod.nextcloud_occ_value
nextcloud_occ_config_set_batch = config_mod.nextcloud_occ_config_set_batch
PREVIOUS_VALUE_EXPRESSION = (
"{{ ( (item.configvalue | to_json) if (item.configvalue is mapping or "
@@ -202,11 +203,56 @@ class TestNextcloudPluginConfigPending(unittest.TestCase):
self.assertEqual(_keys(pending), [("deck", "k")])
class TestNextcloudOccConfigSetBatch(unittest.TestCase):
def test_an_untyped_entry_hands_occ_only_the_value(self):
self.assertEqual(
nextcloud_occ_config_set_batch(
[
{
"appid": "deck",
"configkey": "k",
"type": None,
"value": "-1",
}
]
),
[
{
"argv": ["config:app:set", "deck", "k", "--value", "-1"],
"label": "deck/k",
}
],
)
def test_a_typed_entry_sets_its_type_without_a_prompt(self):
batch = nextcloud_occ_config_set_batch(
[{"appid": "spreed", "configkey": "k", "type": "array", "value": "[]"}]
)
self.assertEqual(
batch[0]["argv"],
[
"config:app:set",
"spreed",
"k",
"--no-interaction",
"--type=array",
"--value",
"[]",
],
)
def test_no_pending_entry_is_no_call(self):
self.assertEqual(nextcloud_occ_config_set_batch([]), [])
class TestFilterModule(unittest.TestCase):
def test_filter_is_registered(self):
self.assertEqual(
config_mod.FilterModule().filters(),
{"nextcloud_plugin_config_pending": nextcloud_plugin_config_pending},
{
"nextcloud_plugin_config_pending": nextcloud_plugin_config_pending,
"nextcloud_occ_config_set_batch": nextcloud_occ_config_set_batch,
},
)
@@ -0,0 +1,198 @@
from __future__ import annotations
import base64
import importlib.util
import json
import re
import shlex
import sys
import unittest
from ansible.plugins.filter.core import b64encode, to_json
from ansible.template import Templar, trust_as_template
from jinja2 import Environment
from utils.cache.files import PROJECT_ROOT
from utils.cache.yaml import load_yaml_any
ROLE = PROJECT_ROOT / "roles/web-app-nextcloud"
STAGES = ROLE / "tasks/02_manager_ops/06_plugin"
INSTALL = STAGES / "02_install.yml"
ENABLE = STAGES / "03_enable.yml"
CONFIGURE = STAGES / "04_configure.yml"
FRESH = {"rc": 0, "stdout": "deck 1.18.6 installed\ndeck enabled", "stderr": ""}
KEPT_DISABLED = {"rc": 0, "stdout": "deck 1.18.6 installed", "stderr": ""}
PLACED = {"rc": 1, "stdout": "recognize already installed", "stderr": ""}
INCOMPATIBLE = {
"rc": 1,
"stdout": "deck is not compatible with this version of the server",
"stderr": "",
}
UNAVAILABLE = {
"rc": 1,
"stdout": "Error: Could not download app deck, it was not found on the appstore",
"stderr": "",
}
def _filters(rel_path: str, name: str) -> dict:
spec = importlib.util.spec_from_file_location(name, str(ROLE / rel_path))
module = importlib.util.module_from_spec(spec)
assert spec and spec.loader
sys.modules[name] = module
spec.loader.exec_module(module)
return module.FilterModule().filters()
ENV = Environment(autoescape=True)
ENV.filters.update(
_filters("filter_plugins/nextcloud_install.py", "nextcloud_install_stages")
)
ENV.filters.update(
_filters("filter_plugins/nextcloud_plugin_config.py", "nextcloud_config_stages")
)
ENV.filters.update({"to_json": to_json, "b64encode": b64encode})
def _tasks(path) -> list[dict]:
return load_yaml_any(str(path), default_if_missing=[])
def _runs(task: dict, install_result: dict) -> bool:
conditions = task["when"] if isinstance(task["when"], list) else [task["when"]]
return all(
bool(ENV.compile_expression(condition)(install_result=install_result))
for condition in conditions
)
def _queue(path) -> dict:
return next(task for task in _tasks(path) if "Queue" in task.get("name", ""))
class TestEnableStageAfterInstall(unittest.TestCase):
def setUp(self):
tasks = _tasks(INSTALL)
self.enable_index = next(
i
for i, task in enumerate(tasks)
if task.get("include_tasks") == "03_enable.yml"
)
self.enable = tasks[self.enable_index]
self.queue_index = next(
i for i, task in enumerate(tasks) if "Queue" in task.get("name", "")
)
self.queue = tasks[self.queue_index]
def test_an_install_that_enabled_the_plugin_skips_the_enable_stage(self):
self.assertFalse(_runs(self.enable, FRESH))
self.assertTrue(_runs(self.queue, FRESH))
def test_an_install_that_left_the_plugin_off_runs_the_enable_stage(self):
for result in (PLACED, KEPT_DISABLED):
with self.subTest(stdout=result["stdout"]):
self.assertTrue(_runs(self.enable, result))
self.assertFalse(_runs(self.queue, result))
def test_a_plugin_that_cannot_run_is_neither_enabled_nor_queued(self):
for result in (INCOMPATIBLE, UNAVAILABLE):
with self.subTest(stdout=result["stdout"]):
self.assertFalse(_runs(self.enable, result))
self.assertFalse(_runs(self.queue, result))
def test_the_direct_queue_appends_like_the_enable_stage(self):
self.assertEqual(
self.queue["ansible.builtin.set_fact"],
_queue(ENABLE)["ansible.builtin.set_fact"],
)
def test_the_queue_keeps_the_install_loop_order(self):
self.assertEqual(self.queue_index, self.enable_index + 1)
OCC = "/var/www/html/occ"
ENTRIES = [
{"appid": "user_ldap", "configkey": "s01ldap_port", "type": None, "value": "389"},
{
"appid": "spreed",
"configkey": "stun_servers",
"type": "array",
"value": '["coturn.example:3481"]',
},
{
"appid": "user_ldap",
"configkey": "s01ldap_group_filter",
"type": None,
"value": "(&(|(objectclass=groupOfNames))(cn=it's $HOME `x`))",
},
{"appid": "richdocuments", "configkey": "wopi_url", "type": None, "value": ""},
{
"appid": "user_ldap",
"configkey": "s01ldap_groupfilter_objectclass",
"type": None,
"value": "groupOfUniqueNames\nposixGroup",
},
{"appid": "whiteboard", "configkey": "jwt_secret_key", "type": None, "value": "é"},
]
def _batch(pending: list[dict]) -> list[dict]:
task = next(
task
for task in _tasks(CONFIGURE)
if "Set the pending plugin configuration" in task.get("name", "")
)
stdin = task["ansible.builtin.shell"]["stdin"]
match = re.fullmatch(
r"<\?php \$occBatch = '\{\{(.+?)\}\}'; \?>"
r"\{\{ lookup\('file', 'php/utils/occ_batch\.php'\) \}\}",
stdin.strip(),
)
assert match, stdin
payload = ENV.compile_expression(match.group(1))(
nextcloud_plugin_config_pending=pending
)
return json.loads(base64.b64decode(payload, validate=True))
PREVIOUS_SHELL_LINE = (
"{{ occ | quote }} config:app:set {{ item.appid | quote }} "
"{{ item.configkey | quote }}"
"{{ (' --no-interaction --type=' ~ item.type) if item.type is not none else '' }}"
" --value {{ item.value | quote }}"
)
def _previous_argv(item: dict) -> list[str]:
"""The argv the former ``sh -s`` line handed to occ for one pending entry.
Args:
item: one ``nextcloud_plugin_config_pending`` entry.
"""
line = Templar(variables={"occ": OCC, "item": item}).template(
trust_as_template(PREVIOUS_SHELL_LINE)
)
return shlex.split(line)
class TestConfigureBatch(unittest.TestCase):
def test_every_pending_entry_reaches_the_runner_in_order(self):
self.assertEqual(
[call["label"] for call in _batch(ENTRIES)],
[f"{item['appid']}/{item['configkey']}" for item in ENTRIES],
)
def test_each_call_hands_occ_the_argv_of_the_former_shell_line(self):
for item, call in zip(ENTRIES, _batch(ENTRIES), strict=True):
with self.subTest(key=item["configkey"]):
self.assertEqual([OCC, *call["argv"]], _previous_argv(item))
def test_an_untyped_entry_keeps_the_stored_type(self):
argv = _batch(ENTRIES[:1])[0]["argv"]
self.assertNotIn("--no-interaction", argv)
self.assertFalse(any(arg.startswith("--type") for arg in argv))
if __name__ == "__main__":
unittest.main()
@@ -23,6 +23,15 @@ def _load_module():
_M = _load_module()
_ARGV = [
"--mail-host",
"mail.example.org",
"--mail-domain",
"example.org",
"--dmarc-rua",
"administrator@example.org",
]
def _txt_rdata(*chunks: str) -> bytes:
out = b""
@@ -188,6 +197,7 @@ def _args(**overrides):
base = {
"mail_host": "mail.example.org",
"mail_domain": "example.org",
"dmarc_rua": "administrator@example.org",
"dkim_selector": "sel1",
"smtp_port": 25,
"timeout": 1.0,
@@ -269,18 +279,68 @@ class TestChecks(unittest.TestCase):
_M.check_spf(_args(), [], self.report)
self.assertEqual(self.report.failures, 1)
def test_dmarc_requires_a_policy(self):
with self._patch_query(
{("_dmarc.example.org", _M.TYPE_TXT): ["v=DMARC1; p=reject"]}
):
_M.check_dmarc(_args(), [], self.report)
def _check_dmarc_record(self, record, **overrides):
with self._patch_query({("_dmarc.example.org", _M.TYPE_TXT): [record]}):
_M.check_dmarc(_args(**overrides), [], self.report)
def test_dmarc_requires_a_policy_and_the_rua_address(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; rua=mailto:administrator@example.org; "
"ruf=mailto:administrator@example.org; adkim=s; aspf=s"
)
self.assertEqual(self.report.failures, 0)
def test_dmarc_without_policy_fails(self):
with self._patch_query({("_dmarc.example.org", _M.TYPE_TXT): ["v=DMARC1;"]}):
_M.check_dmarc(_args(), [], self.report)
self._check_dmarc_record("v=DMARC1; rua=mailto:administrator@example.org")
self.assertEqual(self.report.failures, 1)
def test_dmarc_subdomain_policy_is_not_a_policy(self):
self._check_dmarc_record(
"v=DMARC1; sp=reject; rua=mailto:administrator@example.org"
)
self.assertEqual(self.report.failures, 1)
def test_dmarc_without_rua_fails(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; ruf=mailto:administrator@example.org"
)
self.assertEqual(self.report.failures, 1)
def test_dmarc_rua_to_another_mailbox_fails(self):
self._check_dmarc_record("v=DMARC1; p=reject; rua=mailto:admin@example.org")
self.assertEqual(self.report.failures, 1)
def test_dmarc_rua_matches_among_several_uris_with_size_limits(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; "
"RUA=mailto:reports@elsewhere.net!10m, mailto:Administrator@Example.org!5m"
)
self.assertEqual(self.report.failures, 0)
def test_dmarc_duplicate_rua_invalidates_the_record(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; rua=mailto:reports@elsewhere.net; "
"rua=mailto:administrator@example.org"
)
self.assertEqual(self.report.failures, 1)
def test_dmarc_tags_differing_only_in_case_are_duplicates(self):
with self.assertRaises(ValueError):
_M.dmarc_tags("v=DMARC1; p=reject; P=none")
def test_dmarc_rua_address_is_percent_decoded(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; rua=mailto:dmarc%2Breports@example.org!10m",
dmarc_rua="dmarc+reports@example.org",
)
self.assertEqual(self.report.failures, 0)
def test_dmarc_rua_header_fields_are_not_part_of_the_address(self):
self._check_dmarc_record(
"v=DMARC1; p=reject; rua=mailto:administrator@example.org?subject=dmarc"
)
self.assertEqual(self.report.failures, 0)
def test_dkim_key_must_be_present(self):
record = "v=DKIM1; k=rsa; p=MIIBIjANBg"
with self._patch_query(
@@ -387,20 +447,25 @@ class TestSmtpBanner(unittest.TestCase):
class TestMain(unittest.TestCase):
def test_no_resolver_configured_fails_fast(self):
with mock.patch.object(_M, "system_resolvers", return_value=[]):
rc = _M.main(
["--mail-host", "mail.example.org", "--mail-domain", "example.org"]
)
rc = _M.main(_ARGV)
self.assertEqual(rc, 1)
def test_missing_dmarc_rua_is_a_usage_error(self):
with (
mock.patch.object(_M, "system_resolvers", return_value=["192.0.2.53"]),
mock.patch.object(_M.sys, "stderr"),
self.assertRaises(SystemExit) as raised,
):
_M.main(_ARGV[:4])
self.assertEqual(raised.exception.code, 2)
def test_success_path_returns_zero(self):
clean = _M.Report()
with (
mock.patch.object(_M, "system_resolvers", return_value=["192.0.2.53"]),
mock.patch.object(_M, "run_checks", return_value=clean),
):
rc = _M.main(
["--mail-host", "mail.example.org", "--mail-domain", "example.org"]
)
rc = _M.main(_ARGV)
self.assertEqual(rc, 0)
def test_unreachable_resolver_is_a_finding_not_a_traceback(self):
@@ -409,16 +474,7 @@ class TestMain(unittest.TestCase):
mock.patch.object(_M, "run_checks", side_effect=_M.DnsError("unreachable")),
mock.patch.object(_M.time, "sleep"),
):
rc = _M.main(
[
"--mail-host",
"mail.example.org",
"--mail-domain",
"example.org",
"--retries",
"2",
]
)
rc = _M.main([*_ARGV, "--retries", "2"])
self.assertEqual(rc, 1)
def test_failures_retry_then_report(self):
@@ -429,16 +485,7 @@ class TestMain(unittest.TestCase):
mock.patch.object(_M, "run_checks", return_value=broken) as runner,
mock.patch.object(_M.time, "sleep"),
):
rc = _M.main(
[
"--mail-host",
"mail.example.org",
"--mail-domain",
"example.org",
"--retries",
"3",
]
)
rc = _M.main([*_ARGV, "--retries", "3"])
self.assertEqual(rc, 1)
self.assertEqual(runner.call_count, 3)
@@ -498,18 +545,7 @@ class TestMetricsFile(unittest.TestCase):
mock.patch.object(_M, "run_checks", return_value=broken),
mock.patch.object(_M.time, "sleep"),
):
rc = _M.main(
[
"--mail-host",
"mail.example.org",
"--mail-domain",
"example.org",
"--retries",
"1",
"--metrics-file",
str(target),
]
)
rc = _M.main([*_ARGV, "--retries", "1", "--metrics-file", str(target)])
self.assertEqual(rc, 1)
self.assertIn(
"stalwart_mail_facts_failures",
@@ -525,9 +561,7 @@ class TestMetricsFile(unittest.TestCase):
mock.patch.object(_M, "system_resolvers", return_value=["192.0.2.53"]),
mock.patch.object(_M, "run_checks", return_value=clean),
):
rc = _M.main(
["--mail-host", "mail.example.org", "--mail-domain", "example.org"]
)
rc = _M.main(_ARGV)
self.assertEqual(rc, 0)
self.assertEqual(list(Path(tmp).iterdir()), [])