From 457e58263d5d476bed59acafaa77a5cd342b12c5 Mon Sep 17 00:00:00 2001 From: "infinito-nexus-updater[bot]" <286144613+infinito-nexus-updater[bot]@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:19:40 +0000 Subject: [PATCH 01/11] update(versions): update pins with a declared source (web-app-suitecrm) --- roles/web-app-suitecrm/meta/services.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-suitecrm/meta/services.yml b/roles/web-app-suitecrm/meta/services.yml index 042eb5b315..af98ccff10 100644 --- a/roles/web-app-suitecrm/meta/services.yml +++ b/roles/web-app-suitecrm/meta/services.yml @@ -59,7 +59,7 @@ suitecrm: # nocheck: docker-version version: 8.2-apache custom: true - app_version: 8.9.1 + app_version: 8.10.2 phpredis_version: 6.3.0 update: - key: app_version From d2abd332f0035c8ba7b207e1a42055b7bcafa36c Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Thu, 15 Oct 2026 11:28:55 +0200 Subject: [PATCH 02/11] fix(web-app-suitecrm): let the external-auth provisioning path create its user SuiteCRM 8.10.2 tightened User::hasSaveAccess() from return true to is_admin($current_user) for a record without an id, and restrictAdminOnlyFields() now forces status Inactive on such a record. CreateUser::create() runs while the browser is still anonymous, because it is the LDAP and SAML auto-create path, so every FIRST login of a directory account ends in RuntimeException: Not authorized and HTTP 500 on POST /login. The pre-existing administrator is unaffected, which is why one persona passes and one fails in both auth modes. 451f63cbdd carried the role from 8.9.1 to 8.10.2; 8.9.1 exempted a new record from both guards. The patch raises the admin context around that one save and restores it in a finally, rather than relaxing the guards, which stay closed for every request-driven save. create() has a single caller in its own file, createExternalAuthUser, which sets is_admin 0 and external_auth_only 1 before calling, so the raised context cannot mint an administrator. patch runs with -F0 so a hunk upstream has moved fails the build instead of landing somewhere else. The patch path is derived once in vars/main.yml and reaches the Dockerfile as a build arg, following the role's existing SUITECRM_ENTRYPOINT_SCRIPT_* shape, so the file name is written once. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 495d23dbde0da94d76a6631c6d6fc70d78207e5e) # Conflicts: # roles/web-app-suitecrm/vars/main.yml --- roles/web-app-suitecrm/README.md | 8 +++ roles/web-app-suitecrm/files/Dockerfile | 8 +++ .../patch/external-auth-user-create.patch | 50 +++++++++++++++++++ .../web-app-suitecrm/tasks/01_manager_ops.yml | 8 +++ .../web-app-suitecrm/templates/compose.yml.j2 | 2 + roles/web-app-suitecrm/vars/main.yml | 6 +++ 6 files changed, 82 insertions(+) create mode 100644 roles/web-app-suitecrm/files/patch/external-auth-user-create.patch diff --git a/roles/web-app-suitecrm/README.md b/roles/web-app-suitecrm/README.md index 8091a52906..a4b86839f4 100644 --- a/roles/web-app-suitecrm/README.md +++ b/roles/web-app-suitecrm/README.md @@ -142,6 +142,14 @@ docker run --rm -it \ the Tor SOCKS proxy whenever the issuer is an onion, because a `.onion` issuer is unresolvable directly. `/auth` stays reachable as the local break-glass login. +- **Auto-created accounts** reach `User::save()` through SuiteCRM's own `CreateUser` + while the request is still anonymous. Since 8.10.2 that save answers + `RuntimeException: Not authorized` and forces `status = Inactive`, so + `files/patch/external-auth-user-create.patch` raises an admin context around the save + and restores the caller's afterwards. [`files/Dockerfile`](./files/Dockerfile) applies + it with `patch -F0` on the unpacked release; the image build fails once the hunk stops + matching. + ## Persona contract opt-outs [`templates/env.j2`](./templates/env.j2) seeds only the administrator into SuiteCRM (`SUITECRM_ADMIN_USERNAME` / `SUITECRM_ADMIN_PASSWORD`); `biber` reaches the app through SuiteCRM's own SAML login against Keycloak, where `SAML_AUTO_CREATE` mints the account on first assertion. In the `services.sso.enabled: false` matrix variants there is no SAML entry point and `biber` has no native account, so [`templates/playwright.env.j2`](./templates/playwright.env.j2) renders `PERSONA_BIBER_BLOCKED=true` for the generic persona journeys (the LDAP spec still signs `biber` in from the directory). The `administrator` persona keeps its native login via `ADMIN_NATIVE_PASSWORD` and runs in every variant. diff --git a/roles/web-app-suitecrm/files/Dockerfile b/roles/web-app-suitecrm/files/Dockerfile index 08c8f95e71..a2a6a6ec0e 100644 --- a/roles/web-app-suitecrm/files/Dockerfile +++ b/roles/web-app-suitecrm/files/Dockerfile @@ -19,6 +19,7 @@ RUN rm -rf /var/lib/apt/lists/* \ libonig-dev \ libxml2-dev \ libldap2-dev \ + patch \ && docker-php-ext-configure ldap --with-libdir=lib/x86_64-linux-gnu \ && docker-php-ext-install \ pdo \ @@ -56,6 +57,8 @@ FROM base AS builder ARG SUITECRM_APP_VERSION ARG SUITECRM_APP_DIR ARG SUITECRM_LDAP_EXTENSION_DIR +ARG SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL +ARG SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER WORKDIR ${SUITECRM_APP_DIR} @@ -69,6 +72,11 @@ RUN set -eux; \ rm /tmp/suitecrm.tar.gz; \ chmod +x bin/console +COPY ${SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL} ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER} +RUN set -eux; \ + patch -p1 -F0 -i ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER}; \ + rm ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER} + RUN set -eux; \ composer install \ --prefer-dist \ diff --git a/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch new file mode 100644 index 0000000000..9d00efbcd4 --- /dev/null +++ b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch @@ -0,0 +1,50 @@ +SuiteCRM 8.10.2 tightened User::hasSaveAccess() from `return true` to +`return is_admin($current_user)` for a record without an id, and +restrictAdminOnlyFields() now forces `status = Inactive` on such a record +(release commit 6c7ce002a). CreateUser::create() runs while the browser is +still anonymous, because it is the LDAP and SAML auto-create path +(AppLdapUserProvider::createUser, AppSamlUserFactory::createUser, both via +UserHandler::createExternalAuthUser), so every first login of a directory +account ends in `RuntimeException: Not authorized`, HTTP 500 on POST /login +and no session after a SAML assertion. v8.10.1 and earlier exempted a new +record from both guards; upstream has no release and no hotfix carrying a fix. + +\CreateUser is reachable only from that provisioning stack and from the +installer, so the admin context is raised here instead of relaxing the guards, +which keeps them closed for every request-driven save. + +Generated with `diff -u` against the v8.10.2 tarball and applied with `-F0`, so +the build fails instead of letting GNU patch's default fuzz factor land the +hunk somewhere upstream moved it to. + +--- a/public/legacy/include/portability/User/CreateUser.php ++++ b/public/legacy/include/portability/User/CreateUser.php +@@ -68,6 +68,8 @@ + */ + public function create(string $name, array $fields): ?SugarBean + { ++ global $current_user; ++ + $user = BeanFactory::newBean('Users'); + $user->user_name = $name; + +@@ -77,7 +79,18 @@ + $user->$key = $value; + } + +- $user->save(); ++ $callerUser = $current_user; ++ ++ if (!is_admin($current_user)) { ++ $current_user = BeanFactory::newBean('Users'); ++ $current_user->is_admin = 1; ++ } ++ ++ try { ++ $user->save(); ++ } finally { ++ $current_user = $callerUser; ++ } + + if (empty($user)) { + return null; diff --git a/roles/web-app-suitecrm/tasks/01_manager_ops.yml b/roles/web-app-suitecrm/tasks/01_manager_ops.yml index 373126537c..f0a6bc481a 100644 --- a/roles/web-app-suitecrm/tasks/01_manager_ops.yml +++ b/roles/web-app-suitecrm/tasks/01_manager_ops.yml @@ -30,6 +30,14 @@ - compose-up - compose-build +- name: "πŸš€ Deploy '{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS }}'" + copy: + src: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_SRC }}" + dest: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS }}" + notify: + - compose-up + - compose-build + - name: "🐳 Docker Compose Up for '{{ application_id }}'" meta: flush_handlers diff --git a/roles/web-app-suitecrm/templates/compose.yml.j2 b/roles/web-app-suitecrm/templates/compose.yml.j2 index a4a9f433e8..2735bcb41b 100644 --- a/roles/web-app-suitecrm/templates/compose.yml.j2 +++ b/roles/web-app-suitecrm/templates/compose.yml.j2 @@ -9,6 +9,8 @@ SUITECRM_PHPREDIS_VERSION: "{{ SUITECRM_PHPREDIS_VERSION }}" SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL: "{{ SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL }}" SUITECRM_ENTRYPOINT_SCRIPT_DOCKER: "{{ SUITECRM_ENTRYPOINT_SCRIPT_DOCKER }}" + SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL }}" + SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER }}" SUITECRM_LDAP_EXTENSION_DIR: "{{ SUITECRM_LDAP_EXTENSION_DIR }}" SUITECRM_APP_DIR: "{{ SUITECRM_APP_DIR }}" SUITECRM_PHP_MEMORY_LIMIT: "{{ SUITECRM_PHP_MEMORY_LIMIT }}" diff --git a/roles/web-app-suitecrm/vars/main.yml b/roles/web-app-suitecrm/vars/main.yml index ec66f96ded..dbec4594ec 100644 --- a/roles/web-app-suitecrm/vars/main.yml +++ b/roles/web-app-suitecrm/vars/main.yml @@ -41,5 +41,11 @@ SUITECRM_ENTRYPOINT_SCRIPT_HOST_ABS: "{{ [ lookup('container', application_id, ' SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL: "volumes/{{ SUITECRM_ENTRYPOINT_SCRIPT_FILE }}" SUITECRM_ENTRYPOINT_SCRIPT_DOCKER: "{{ [ '/usr/local/bin/', SUITECRM_ENTRYPOINT_SCRIPT_FILE ] | path_join }}" +SUITECRM_EXTERNAL_AUTH_PATCH_FILE: "external-auth-user-create.patch" +SUITECRM_EXTERNAL_AUTH_PATCH_SRC: "patch/{{ SUITECRM_EXTERNAL_AUTH_PATCH_FILE }}" +SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS: "{{ [ lookup('container', application_id, 'directories.volumes'), SUITECRM_EXTERNAL_AUTH_PATCH_FILE ] | path_join }}" +SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL: "volumes/{{ SUITECRM_EXTERNAL_AUTH_PATCH_FILE }}" +SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER: "{{ [ '/tmp/', SUITECRM_EXTERNAL_AUTH_PATCH_FILE ] | path_join }}" + SUITECRM_MAX_OLD_SPACE_SIZE: "{{ lookup('node_max_old_space_size', application_id, entity_name) }}" SUITECRM_PHP_MEMORY_LIMIT: "{{ lookup('service_memory_mb', application_id, entity_name, 0.25) }}M" From 086bbaf8a3bd66b789835f2194ef258deb32812a Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 14:18:23 +0200 Subject: [PATCH 03/11] fix(web-app-suitecrm): raise the stored system user around the external-auth save 495d23dbde raised an admin context with a fresh in-memory bean, which fixed hasSaveAccess() and moved the failure one guard further. User::verify_data() compares $this->id == $current_user->id, and between two unsaved beans that is '' == '', so a deployment with a single administrator takes the last-admin branch for the very account CreateUser is in the middle of creating. The branch reads the legacy $mod_strings, which this request never loads. That is why the two SuiteCRM jobs of CI run 38818071194 looked like two failures: the LDAP variant ended in ErrorException "Trying to access array offset on value of type null" at modules/Users/User.php:1678 and answered HTTP 500 to POST /login, while the SAML variant took the same verify_data() failure, avoided the 500 because $mod_strings happened to be loaded, redirected to index.php?action=Error&module=Users and left no session, so its spec timed out waiting for the SPA instead. Both are the first login of a persona that has no local row yet; administrator passes in both because it already has one. Use the stored system user, which has a real id, so hasSaveAccess() passes, restrictAdminOnlyFields() early-returns, setIsAdmin()'s reloaded bean resolves and '' == '1' skips the last-admin branch. Assign $current_user before calling getSystemUser(), not from its return value. User::retrieve() runs loadPreferences() after filling the bean, and both UserPreference::loadPreferences() and reloadPreferences() dereference $current_user->id unguarded, so retrieving while the caller's anonymous $current_user is still in place raises "Attempt to read property id on null" and answers HTTP 500 one line further on. Assigning first also means the only session preferences key written is the system user's own, never the key of the account signing in. The patch header records both failure modes, because the patch outlives this commit and is applied at image build time with -F0, so whoever reads it next when a hunk stops matching needs to know what it was for. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 207cdc70ff24c46afcdd009c39e3c111c9405ab2) --- roles/web-app-suitecrm/README.md | 8 +++---- .../patch/external-auth-user-create.patch | 21 ++++++++++++++++++- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/roles/web-app-suitecrm/README.md b/roles/web-app-suitecrm/README.md index a4b86839f4..a23cea4fa5 100644 --- a/roles/web-app-suitecrm/README.md +++ b/roles/web-app-suitecrm/README.md @@ -145,10 +145,10 @@ docker run --rm -it \ - **Auto-created accounts** reach `User::save()` through SuiteCRM's own `CreateUser` while the request is still anonymous. Since 8.10.2 that save answers `RuntimeException: Not authorized` and forces `status = Inactive`, so - `files/patch/external-auth-user-create.patch` raises an admin context around the save - and restores the caller's afterwards. [`files/Dockerfile`](./files/Dockerfile) applies - it with `patch -F0` on the unpacked release; the image build fails once the hunk stops - matching. + `files/patch/external-auth-user-create.patch` makes the stored system user the current + user around the save and restores the caller's afterwards. + [`files/Dockerfile`](./files/Dockerfile) applies it with `patch -F0` on the unpacked + release; the image build fails once the hunk stops matching. ## Persona contract opt-outs diff --git a/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch index 9d00efbcd4..6f7275b53d 100644 --- a/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch +++ b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch @@ -13,6 +13,25 @@ record from both guards; upstream has no release and no hotfix carrying a fix. installer, so the admin context is raised here instead of relaxing the guards, which keeps them closed for every request-driven save. +The raised context is the stored system user, never a fresh in-memory bean +carrying `is_admin = 1`. User::verify_data() compares `$this->id == +$current_user->id`, which is `'' == ''` between two unsaved beans, so a +deployment holding a single administrator takes the last-admin branch for the +very account it is creating. That branch reads the legacy `$mod_strings`, which +this request never loads, so an LDAP login ends in `ErrorException: Trying to +access array offset on value of type null` at modules/Users/User.php:1678 and +HTTP 500 on POST /login, while a SAML assertion redirects to +`index.php?action=Error&module=Users` and leaves no session. + +`$current_user` is assigned before `getSystemUser()` runs, not from its return +value. User::retrieve() calls loadPreferences() after it has filled the bean, +and both UserPreference::loadPreferences() and reloadPreferences() dereference +`$current_user->id` unguarded, so retrieving while the caller's anonymous +`$current_user` is still in place would raise `Attempt to read property "id" on +null` and answer HTTP 500 again. Assigning first makes the global the very bean +being filled, so those reads see an object throughout and the only session key +written is the system user's own. + Generated with `diff -u` against the v8.10.2 tarball and applied with `-F0`, so the build fails instead of letting GNU patch's default fuzz factor land the hunk somewhere upstream moved it to. @@ -37,7 +56,7 @@ hunk somewhere upstream moved it to. + + if (!is_admin($current_user)) { + $current_user = BeanFactory::newBean('Users'); -+ $current_user->is_admin = 1; ++ $current_user->getSystemUser(); + } + + try { From dc3d30acaa87a009dbb81e5c827b8f81bc32c5d5 Mon Sep 17 00:00:00 2001 From: Prageeth Panicker Date: Fri, 16 Oct 2026 14:41:05 +0000 Subject: [PATCH 04/11] fix(sys-ctl-hlth-journalctl): skip container logs in the host journal check Problem: Every compose service logs to journald through sys-svc-container/templates/base.yml.j2. The daily journal health check grepped the whole journal for "error", so routine application errors from containers triggered the host alert (email/Telegram) and buried real host problems. Change: - Read the last day of the journal as JSON (MESSAGE, CONTAINER_ID, SYSLOG_IDENTIFIER, _SYSTEMD_UNIT) and skip every entry that carries a CONTAINER_ID field. Docker daemon and system errors are still reported. - Report each match as one readable "source: message" line instead of raw journal output. - Stream the journalctl output line by line instead of buffering a whole day of logs in memory. - Exit 2 with journalctl's stderr if journalctl itself fails, instead of silently reporting "no errors". - Rewrite the check in Python (files/python/script.py, deployed by sys-service like the other sys-ctl-hlth roles) and decode non-UTF-8 messages. - Fix the success message, which claimed "All docker containers are healthy". Impact: Container log errors no longer raise this host alert on any installation. Container health stays covered by sys-ctl-hlth-container; application log analysis belongs in a log platform such as Wazuh. system_service_suppress_flush is unchanged until a deploy confirms that no host errors remain. Tests: Unit tests run the script against a fake journalctl (8 cases): container error ignored, daemon error reported, identifier fallback, non-UTF-8 message, journalctl failure, many container lines streamed past, clean journal passes. The core cases fail against the previous script. --- roles/sys-ctl-hlth-journalctl/README.md | 10 +- .../files/python/script.py | 84 +++++++++++ .../files/shell/script.sh | 10 -- .../roles/sys-ctl-hlth-journalctl/__init__.py | 0 .../sys-ctl-hlth-journalctl/files/__init__.py | 0 .../files/test_script.py | 133 ++++++++++++++++++ 6 files changed, 224 insertions(+), 13 deletions(-) create mode 100755 roles/sys-ctl-hlth-journalctl/files/python/script.py delete mode 100644 roles/sys-ctl-hlth-journalctl/files/shell/script.sh create mode 100644 tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py create mode 100644 tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py create mode 100644 tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py diff --git a/roles/sys-ctl-hlth-journalctl/README.md b/roles/sys-ctl-hlth-journalctl/README.md index c60c77d27d..507b466be1 100644 --- a/roles/sys-ctl-hlth-journalctl/README.md +++ b/roles/sys-ctl-hlth-journalctl/README.md @@ -2,11 +2,13 @@ ## Description -Scans `journalctl` over the last day for β€œerror” entries and alerts if any are found. +Scans `journalctl` over the last day for β€œerror” entries outside container logs and alerts if any are found. ## Overview This role searches the systemd journal for errors over the past day and alerts if any are found. +Container output is skipped, because every compose service logs to journald and application errors would otherwise trigger the host check. +Docker daemon and system errors are still reported. ## Cosmos @@ -23,8 +25,10 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional ( ## Features -- Runs `journalctl --since '1 day ago' | grep -i error`. -- Exits non-zero on matches. +- Reads the last day of the journal as JSON and drops every entry that carries a `CONTAINER_ID` field. +- Matches the remaining messages case-insensitively against `error`. +- Prints one `source: message` line per match, naming the systemd unit or, without one, the syslog identifier, so alerts show who logged the error. +- Exits non-zero on matches, and also when `journalctl` itself fails, so a broken check never passes silently. - Scheduled via systemd timer. - Alerts via `sys-ctl-alm-compose` on detection. diff --git a/roles/sys-ctl-hlth-journalctl/files/python/script.py b/roles/sys-ctl-hlth-journalctl/files/python/script.py new file mode 100755 index 0000000000..cf0426d3ab --- /dev/null +++ b/roles/sys-ctl-hlth-journalctl/files/python/script.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +import json +import subprocess +import sys +import tempfile + +JOURNALCTL = [ + "journalctl", + "--since", + "1 day ago", + "--no-pager", + "--output=json", + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", +] + + +def message_text(value): + """Return a journal MESSAGE as one line of text. + + Args: + value: the MESSAGE field; journald encodes non-UTF-8 messages as a list of bytes. + + Returns: + The message with line breaks collapsed to spaces. + """ + if isinstance(value, list): + value = bytes(value).decode("utf-8", errors="replace") + return " ".join(str(value).splitlines()) + + +def host_errors(lines): + """Yield `source: message` for every non-container entry that mentions an error. + + Args: + lines: iterable of journalctl JSON lines, one entry per line, consumed lazily. + + Returns: + Iterator of readable error lines. + """ + for line in lines: + if not line.strip(): + continue + entry = json.loads(line) + if "CONTAINER_ID" in entry: + continue + message = message_text(entry.get("MESSAGE", "")) + if "error" not in message.lower(): + continue + source = ( + entry.get("_SYSTEMD_UNIT") or entry.get("SYSLOG_IDENTIFIER") or "unknown" + ) + yield f"{source}: {message}" + + +def main(): + print("Checking journalctl for error messages outside container logs...") + # stderr goes to a file, not a pipe: an unread stderr pipe that fills up would block journalctl while stdout is still being read. + with tempfile.TemporaryFile( + mode="w+", encoding="utf-8", errors="replace" + ) as stderr: + with subprocess.Popen( + JOURNALCTL, + stdout=subprocess.PIPE, + stderr=stderr, + encoding="utf-8", + errors="replace", + ) as journal: + errors = list(host_errors(journal.stdout)) + if journal.returncode != 0: + stderr.seek(0) + print( + f"journalctl failed with exit code {journal.returncode}: {stderr.read().strip()}" + ) + return 2 + if errors: + print("Some errors were found:") + print("\n".join(errors)) + return 1 + print("No errors found in the journal.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh b/roles/sys-ctl-hlth-journalctl/files/shell/script.sh deleted file mode 100644 index 4903b597d1..0000000000 --- a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -echo "Checking journalctl for error messages..." -journalctl_errors="$(journalctl --since '1 day ago' --no-pager | grep -i 'error')" -if [ -n "$journalctl_errors" ] -then - echo "Some errors where found: $journalctl_errors" - exit 1 -fi -echo "All docker containers are healthy." -exit 0 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py new file mode 100644 index 0000000000..21c17f01c3 --- /dev/null +++ b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py @@ -0,0 +1,133 @@ +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from utils import PROJECT_ROOT + +SCRIPT = PROJECT_ROOT / "roles/sys-ctl-hlth-journalctl/files/python/script.py" + +JOURNALCTL_STUB = """#!/usr/bin/env bash +printf '%s\\n' "$*" >"${STUB_DIR}/args" +cat "${STUB_DIR}/journal" +if [ "${STUB_RC}" -ne 0 ]; then +\techo "journalctl: unrecognized option '--output-fields'" >&2 +fi +exit "${STUB_RC}" +""" + +APP_ERROR = { + "MESSAGE": "ERROR: pixelfed queue failed", + "CONTAINER_ID": "0123456789ab", + "SYSLOG_IDENTIFIER": "pixelfed", + "_SYSTEMD_UNIT": "docker.service", +} +DAEMON_ERROR = { + "MESSAGE": 'level=error msg="failed to start container"', + "SYSLOG_IDENTIFIER": "dockerd", + "_SYSTEMD_UNIT": "docker.service", +} +KERNEL_ERROR = {"MESSAGE": "EXT4-fs error (device sda1)", "SYSLOG_IDENTIFIER": "kernel"} +BINARY_ERROR = { + "MESSAGE": list(b"I/O error \xff on disk"), + "_SYSTEMD_UNIT": "smartd.service", +} +SYSTEM_INFO = { + "MESSAGE": "Started Daily apt download activities.", + "_SYSTEMD_UNIT": "init.scope", +} + + +class TestHlthJournalctlScript(unittest.TestCase): + def run_script(self, *entries, rc=0): + """Run the health script against a stubbed journalctl. + + Args: + entries: journal entries the stub prints, one JSON object per line. + rc: exit code of the stubbed journalctl. + + Returns: + Tuple of the exit code, the script output, and the journalctl arguments. + """ + with tempfile.TemporaryDirectory() as tmp: + bin_dir = Path(tmp) + stub = bin_dir / "journalctl" + stub.write_text(JOURNALCTL_STUB) + stub.chmod(0o755) + (bin_dir / "journal").write_text( + "".join(json.dumps(entry) + "\n" for entry in entries) + ) + env = dict(os.environ) + env["PATH"] = f"{bin_dir}{os.pathsep}{env['PATH']}" + env["STUB_DIR"] = tmp + env["STUB_RC"] = str(rc) + done = subprocess.run( + [sys.executable, str(SCRIPT)], + env=env, + capture_output=True, + text=True, + check=False, + ) + args = (bin_dir / "args").read_text() + return done.returncode, done.stdout, args + + def test_reads_the_journal_as_json_with_the_source_fields(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--output=json", args) + self.assertIn( + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", + args, + ) + + def test_an_application_error_from_a_container_is_ignored(self): + code, _, _ = self.run_script(APP_ERROR, SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_docker_daemon_error_is_reported_with_its_unit(self): + code, output, _ = self.run_script(APP_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn( + 'docker.service: level=error msg="failed to start container"', output + ) + self.assertNotIn("pixelfed queue failed", output) + + def test_an_entry_without_unit_is_named_by_its_identifier(self): + code, output, _ = self.run_script(KERNEL_ERROR) + self.assertEqual(code, 1) + self.assertIn("kernel: EXT4-fs error (device sda1)", output) + + def test_a_non_utf8_message_is_decoded_and_reported(self): + code, output, _ = self.run_script(BINARY_ERROR) + self.assertEqual(code, 1) + self.assertIn("smartd.service: I/O error", output) + + def test_many_container_errors_are_streamed_past_and_not_reported(self): + container_lines = [ + {**APP_ERROR, "MESSAGE": f"ERROR: pixelfed request {n} failed"} + for n in range(20000) + ] + code, output, _ = self.run_script(*container_lines, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertNotIn("pixelfed request", output) + self.assertEqual( + output.splitlines()[-1], + 'docker.service: level=error msg="failed to start container"', + ) + self.assertEqual(len(output.splitlines()), 3) + + def test_a_journal_without_errors_passes(self): + code, _, _ = self.run_script(SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_failing_journalctl_fails_the_check(self): + code, output, _ = self.run_script(rc=1) + self.assertEqual(code, 2) + self.assertIn("journalctl failed with exit code 1", output) + self.assertIn("unrecognized option '--output-fields'", output) + + +if __name__ == "__main__": + unittest.main() From a58abd470c400d171876a67b247300b37f5a4833 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 20:53:12 +0200 Subject: [PATCH 05/11] fix(cache): re-park the applications carrier after every nested app load Job 116975298820 (Nextcloud#5, swarm, arch, btrfs, main 463df06202) hit its budget (rc=124 after 20136 s). The app phase loads every service as a nested `tasks/utils/load/app.yml` include under sys-service-loader, so `tasks/utils/setup/role.yml` parks no carrier between `svc-virt-kata` (L6073) and `update` (L38213). The only re-park in that stretch is the one in `roles/sys-token-store/tasks/write.yml`, and Stalwart's user management reaches it on the manager alone. After its last token write (L23574) the manager carried the new key while both workers kept the old one and rendered the full tree on every task: 85 non-wait tasks of 30 s or more, 63 min from Keycloak (L25641) to the carry past `update` (L38237). Park the carrier after every nested load on every host of the include. A re-park whose key still matches serves the carried payload without rendering. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../contributing/artefact/files/plugins/lookup/applications.md | 2 +- plugins/action/applications_carrier.py | 3 ++- tasks/utils/load/app.yml | 3 +++ utils/cache/README.md | 2 +- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/contributing/artefact/files/plugins/lookup/applications.md b/docs/contributing/artefact/files/plugins/lookup/applications.md index fdca9bd70f..c40f2e9d36 100644 --- a/docs/contributing/artefact/files/plugins/lookup/applications.md +++ b/docs/contributing/artefact/files/plugins/lookup/applications.md @@ -36,7 +36,7 @@ Ansible forks a fresh worker process for every task. The in-process cache inside The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY`, `SYSTEM_EMAIL_DOMAIN` or `group_names`, or a changed path or content of the tokens file). - The stage-wide task sits in `tasks/stages/01_constructor.yml` after the `networks`, `design` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play. -- The carrier is parked again after every role in `tasks/utils/setup/role.yml` and after every persisted token in `roles/sys-token-store/tasks/write.yml`. A re-park whose key still matches serves the carried payload without rendering. +- The carrier is parked again after every role in `tasks/utils/setup/role.yml`, after every nested application load in `tasks/utils/load/app.yml`, and after every persisted token in `roles/sys-token-store/tasks/write.yml`. The first two run on every host of the include, also on hosts the loaded role skips. A re-park whose key still matches serves the carried payload without rendering. - The tokens file enters the key as the SHA-256 of its content, memoised per inode, mtime and size, so a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes. - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`. - User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which read the users cache fresh. diff --git a/plugins/action/applications_carrier.py b/plugins/action/applications_carrier.py index 93c479ff3d..bf4b59c1c2 100644 --- a/plugins/action/applications_carrier.py +++ b/plugins/action/applications_carrier.py @@ -21,7 +21,8 @@ longer match is rejected and the reader renders afresh:: applications_carrier: clear: true -``tasks/utils/setup/role.yml`` parks one after every role and +``tasks/utils/setup/role.yml`` parks one after every role, +``tasks/utils/load/app.yml`` one after every nested application load and ``roles/sys-token-store/tasks/write.yml`` one after every persisted token; where the key still matches they serve the carried payload without rendering. diff --git a/tasks/utils/load/app.yml b/tasks/utils/load/app.yml index abf1e21cb4..a4f2265df1 100644 --- a/tasks/utils/load/app.yml +++ b/tasks/utils/load/app.yml @@ -15,3 +15,6 @@ domain: "{{ lookup('domain', load_app_id) }}" when: - groups[load_app_id] is not defined or inventory_hostname in groups[load_app_id] + +- name: "🧊 Carry the applications SPOT past {{ load_app_id }}" + applications_carrier: {} diff --git a/utils/cache/README.md b/utils/cache/README.md index 7e42126045..d602b889cd 100644 --- a/utils/cache/README.md +++ b/utils/cache/README.md @@ -26,7 +26,7 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read. -Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. +Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. ## Test-Only Helpers πŸ§ͺ From d9126645385d7ee8b5381abed6a420eb2b3a3753 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 20:58:41 +0200 Subject: [PATCH 06/11] fix(web-app-fider): put the design logo into the object store With BLOB_STORAGE=s3 Fider reads blobs only from its bucket, so the logo the role wrote into the blobs table never showed: the header had no logo, the favicon stayed upstream's and the sign-in dialog loaded its image from login.fider.io. The role now writes tenants//logos/infinito-design-logo.png into its bucket from inside the SeaweedFS container (credentials on stdin, curl --aws-sigv4, md5 compare so a repeated run reports UNCHANGED) and sets logo_bkey through the new logo_stored argument. With design or logo off it removes only its own key. The SeaweedFS storage check now proves the bucket write with a post attachment and deletes the post again. It used to upload a tenant logo, which replaces logo_bkey; the role sets that key only while it is empty, so the design logo would never have come back. changed_when compares the script output exactly, because a search for CHANGED also matches UNCHANGED. Co-Authored-By: Claude Opus 5.5 (1M context) --- roles/web-app-fider/TODO.md | 1 - .../files/playwright/test-design.js | 18 ++- .../files/playwright/test-seaweedfs.js | 127 ++++++------------ .../files/shell/design-logo-object.sh | 32 +++++ .../web-app-fider/files/sql/apply-design.sql | 4 +- .../files/sql/select-tenant-id.sql | 2 + roles/web-app-fider/tasks/02_design.yml | 31 ++++- roles/web-app-fider/vars/main.yml | 5 +- 8 files changed, 124 insertions(+), 96 deletions(-) create mode 100644 roles/web-app-fider/files/shell/design-logo-object.sh create mode 100644 roles/web-app-fider/files/sql/select-tenant-id.sql diff --git a/roles/web-app-fider/TODO.md b/roles/web-app-fider/TODO.md index 17f69111c3..c5eb7813a8 100644 --- a/roles/web-app-fider/TODO.md +++ b/roles/web-app-fider/TODO.md @@ -1,4 +1,3 @@ # TODO - With `services.sso.enabled` false the role creates no tenant: `tasks/00_core.yml` includes `tasks/01_setup_oidc.yml` (tenant bootstrap, administrator, activation, provider row) only under `FIDER_OIDC_ENABLED`. Upstream then redirects every route to its installation form `/signup`, which any visitor can submit (read from the task conditions and from `app/middlewares/tenant.go` and `app/handlers/signup.go` of the deployed commit `67c52826`; the compose stack of the design pass ran with `sso` on). Fider has no password sign-in, so such a deployment also has no account. Options: bootstrap tenant and administrator in every variant and keep only the provider row under the flag; or refuse `/signup` at the proxy while no tenant exists. -- With the object store enabled (`BLOB_STORAGE=s3`) the board keeps upstream's favicon and shows no logo, and the sign-in dialog loads `https://login.fider.io/static/assets/logo.png` from the browser (measured on the compose stack before a logo was set: the image rendered in the dialog). Cause: `tasks/02_design.yml` places the logo in the `blobs` table, which the S3 blob store never reads, so `FIDER_DESIGN_LOGO_ENABLED` is false there and `tenants.logo_bkey` stays empty; upstream's upload route `POST /_api/admin/settings/general` needs a session cookie. Options: put the object `tenants//logos/infinito-design-logo.png` into the role's bucket with the role's object store credentials and set `logo_bkey`; or upload through the app's API with a session obtained from the identity provider. diff --git a/roles/web-app-fider/files/playwright/test-design.js b/roles/web-app-fider/files/playwright/test-design.js index f7ecf23544..f2ec357305 100644 --- a/roles/web-app-fider/files/playwright/test-design.js +++ b/roles/web-app-fider/files/playwright/test-design.js @@ -9,7 +9,7 @@ const { galleryEnabled, tokenValue, } = require("./design"); -const { decodeDotenvQuotedValue, gotoOnion, performKeycloakLoginForm } = require("./personas"); +const { apiGetOnion, decodeDotenvQuotedValue, gotoOnion, performKeycloakLoginForm } = require("./personas"); const { isServiceEnabled, skipUnlessServiceEnabled } = require("./service-gating"); const { resolveTimeout } = require("./timeouts"); const { clickFiderSsoButton } = require("./_shared"); @@ -17,8 +17,6 @@ const { clickFiderSsoButton } = require("./_shared"); const base = decodeDotenvQuotedValue(process.env.FIDER_BASE_URL || "").replace(/\/$/, ""); const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); -const title = decodeDotenvQuotedValue(process.env.DESIGN_TITLE || ""); -const logoEnabled = process.env.DESIGN_LOGO_ENABLED === "true"; const MODES = ["light", "dark"]; const READY_MS = 60_000; @@ -270,14 +268,18 @@ exports.register = function () { }); test("design: the board carries the configured title", async ({ page }) => { + const title = decodeDotenvQuotedValue(process.env.DESIGN_TITLE || ""); + test.skip(!title, "title replacement is disabled for this role"); await openHome(page); - expect(title, "DESIGN_TITLE must be set in the Playwright env file").toBeTruthy(); await expect(page).toHaveTitle(title); await expect(page.locator(`${BRAND} h1`)).toHaveText(title); }); - test("design: logo and favicon are the configured ones", async ({ page }) => { - test.skip(!logoEnabled, "the role places the logo in the database blob store only; with the object store enabled the tenant keeps its own logo"); + test("design: logo and favicon are the configured ones", async ({ page, request }) => { + test.skip(process.env.DESIGN_LOGO_ENABLED !== "true", "logo replacement is disabled for this role"); + const image = await apiGetOnion(request, `${base}/static/images/logos/infinito-design-logo.png?size=100`); + expect(image.status(), "the logo blob is served from the active blob store").toBe(200); + expect(image.headers()["content-type"]).toBe("image/png"); await openHome(page); const logo = page.locator(`${BRAND} img`); await expect(logo).toBeVisible(); @@ -288,6 +290,10 @@ exports.register = function () { const brandBox = await page.locator(BRAND).boundingBox(); expect(brandBox.width, "logo and title form a wide brand").toBeGreaterThan(brandBox.height); await expect(page.locator("link[rel='icon']").first()).toHaveAttribute("href", LOGO_FAVICON); + await page.locator(PRIMARY).first().click(); + await modalOpened(page); + await expect(page.locator(`${MODAL} img`).first()).toHaveAttribute("src", LOGO_IMAGE); + await expect(page.locator(`${MODAL} img[src*='login.fider.io']`)).toHaveCount(0); }); test("design: the board follows the browser until the switch is used, then the switch wins", async ({ page }) => { diff --git a/roles/web-app-fider/files/playwright/test-seaweedfs.js b/roles/web-app-fider/files/playwright/test-seaweedfs.js index f87b8a456f..58178ba1b3 100644 --- a/roles/web-app-fider/files/playwright/test-seaweedfs.js +++ b/roles/web-app-fider/files/playwright/test-seaweedfs.js @@ -1,16 +1,3 @@ -// SeaweedFS object-store scenario for Fider. -// -// Fider is configured with the S3 blob backend (env.j2: BLOB_STORAGE=s3 + -// BLOB_STORAGE_S3_BUCKET / _ACCESS_KEY_ID / _SECRET_ACCESS_KEY / -// _ENDPOINT_URL / _REGION), so a tenant logo the administrator uploads on the -// admin General settings page is written to the consumer bucket as a new -// private blob (Fider stores blobs with ACL=private and reverse-proxies them -// through its own /images route, which is why the role declares no public -// flag). The action signs the administrator in through Fider's SSO button + -// Keycloak form, opens the admin General settings page, sets the "Your Logo" -// file input to a small PNG and clicks Save; the shared check proves the -// bucket grew via the Filer UI. -// // Required env (rendered by templates/playwright.env.j2): // FIDER_BASE_URL, the OIDC login vars consumed by performKeycloakLoginForm, // and the SEAWEEDFS_* keys consumed by runSeaweedfsStorageCheck. @@ -19,34 +6,21 @@ const { test, expect } = require("@playwright/test"); const { resolveTimeout, isOnionTarget } = require("./timeouts"); const { skipUnlessServiceEnabled } = require("./service-gating"); const { runSeaweedfsStorageCheck, performKeycloakLoginForm, decodeDotenvQuotedValue } = require("./personas"); +const { clickFiderSsoButton } = require("./_shared"); -const LOGO_PNG = Buffer.from( +const ATTACHMENT_PNG_BASE64 = "iVBORw0KGgoAAAANSUhEUgAAAMgAAADICAIAAAAiOjnJAAABeElEQVR42u3SMQ0AAAgEsVeHMDQhEBMM" + - "DE2q4HKpHjgXCTAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + - "scBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAsjIWxwFgYC2OBsTAWxgJjYSyMBcbC" + - "WBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyFsVTAWBgLY4GxMBbGAmNhLIwFxsJY" + - "GAuMhbEwFhgLY2EsMBbGwlhgLIyFscBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAs" + - "jIWxwFgYC2OBsTAWxgJjYSyMBcbCWBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyF" + - "sVTAWBgLY4GxMBbGAmNhLIwFxsJYGAtjqYCxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + - "scBYGAtjgbH4ZgHTqvyKw+KRzwAAAABJRU5ErkJggg==", - "base64", -); - -async function clickFiderSsoButton(locator) { - const signInLink = locator - .getByRole("button", { name: /sign in/i }) - .or(locator.getByRole("link", { name: /sign in/i })); - await signInLink.first().waitFor({ state: "visible", timeout: resolveTimeout(30_000) }); - await signInLink.first().click(); - - const ssoButton = locator.getByRole("link", { name: /continue with/i }); - await ssoButton.first().waitFor({ state: "visible", timeout: resolveTimeout(15_000) }); - await ssoButton.first().click({ force: true, timeout: resolveTimeout(30_000) }); -} + "DE2q4HKpHjgXCTAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + + "scBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAsjIWxwFgYC2OBsTAWxgJjYSyMBcbC" + + "WBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyFsVTAWBgLY4GxMBbGAmNhLIwFxsJY" + + "GAuMhbEwFhgLY2EsMBbGwlhgLIyFscBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAs" + + "jIWxwFgYC2OBsTAWxgJjYSyMBcbCWBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyF" + + "sVTAWBgLY4GxMBbGAmNhLIwFxsJYGAtjqYCxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + + "scBYGAtjgbH4ZgHTqvyKw+KRzwAAAABJRU5ErkJggg=="; test.use({ ignoreHTTPSErrors: true }); -test("seaweedfs: an uploaded Fider tenant logo is stored in the SeaweedFS bucket", async ({ page, browser }) => { +test("seaweedfs: a Fider post attachment is stored in the SeaweedFS bucket", async ({ page, browser }) => { test.skip(isOnionTarget(), "SeaweedFS filer UI is not a Tor surface on an onion node (headless backend)"); skipUnlessServiceEnabled("seaweedfs"); test.setTimeout(resolveTimeout(180_000)); @@ -54,67 +28,52 @@ test("seaweedfs: an uploaded Fider tenant logo is stored in the SeaweedFS bucket const fiderBaseUrl = decodeDotenvQuotedValue(process.env.FIDER_BASE_URL || ""); const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); + const marker = `Storage check ${Date.now()}`; + let post = null; await runSeaweedfsStorageCheck(page, browser, { - label: "a Fider admin tenant-logo upload", - expectInPlaceOverwrite: true, + label: "a Fider post attachment", action: async (appPage) => { expect(fiderBaseUrl, "FIDER_BASE_URL must be set for the Fider seaweedfs check").toBeTruthy(); expect(adminUsername, "ADMIN_USERNAME must be set for the Fider seaweedfs check").toBeTruthy(); expect(adminPassword, "ADMIN_PASSWORD must be set for the Fider seaweedfs check").toBeTruthy(); - const baseUrl = fiderBaseUrl.replace(/\/$/, ""); - - await appPage.goto(`${baseUrl}/`, { waitUntil: "domcontentloaded" }); + await appPage.goto(`${fiderBaseUrl.replace(/\/$/, "")}/`, { waitUntil: "domcontentloaded" }); await clickFiderSsoButton(appPage); await performKeycloakLoginForm(appPage, adminUsername, adminPassword); - await expect( appPage.locator(".c-menu-user").first(), - "the administrator must be authenticated before opening the admin settings", + "the administrator must be authenticated before creating the post", ).toBeVisible({ timeout: resolveTimeout(60_000) }); - await appPage.goto(`${baseUrl}/admin`, { waitUntil: "domcontentloaded" }); - - const fileInput = appPage.locator('input[type="file"][accept="image/*"]').first(); - await expect( - fileInput, - "the Fider admin General settings page must expose the 'Your Logo' image input", - ).toBeAttached({ timeout: resolveTimeout(60_000) }); - - const marker = `infinito-storage-check-${Date.now()}.png`; - await fileInput.setInputFiles({ - name: marker, - mimeType: "image/png", - buffer: LOGO_PNG, - }); - - await expect( - appPage.locator('.c-image-upload img[src^="data:image"], .preview img[src^="data:image"]').first(), - "the selected logo must be loaded into the form (preview rendered) before saving", - ).toBeVisible({ timeout: resolveTimeout(30_000) }); - - const saveButton = appPage.getByRole("button", { name: /^save$/i }).first(); - await expect( - saveButton, - "the Fider admin General settings page must expose a Save action that persists the logo", - ).toBeVisible({ timeout: resolveTimeout(60_000) }); - const [settingsResp] = await Promise.all([ - appPage.waitForResponse( - (r) => /\/_api\/admin\/settings\/general/.test(r.url()) && r.request().method() === "POST", - { timeout: resolveTimeout(60_000) }, - ), - saveButton.click(), - ]); - expect( - settingsResp.ok(), - `Fider must accept the General-settings save that persists the logo (got HTTP ${settingsResp.status()})`, - ).toBeTruthy(); - const reqBody = settingsResp.request().postData() || ""; - expect( - reqBody, - "the General-settings save must carry the logo upload payload (empty upload = silent no-op)", - ).toMatch(/"upload"\s*:\s*\{[^}]*"content"/); + post = await appPage.evaluate( + async ({ title, content }) => { + const response = await fetch("/api/v1/posts", { + method: "POST", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + credentials: "same-origin", + body: JSON.stringify({ + title, + description: "A post that proves attachments reach the object store.", + attachments: [{ upload: { fileName: "storage-check.png", contentType: "image/png", content } }], + }), + }); + return { status: response.status, body: await response.json().catch(() => ({})) }; + }, + { title: marker, content: ATTACHMENT_PNG_BASE64 }, + ); + expect(post.status, "Fider must accept the post with its attachment").toBe(200); }, }); + + const removal = await page.evaluate(async (number) => { + const response = await fetch(`/api/v1/posts/${number}`, { + method: "DELETE", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + credentials: "same-origin", + body: JSON.stringify({ text: "" }), + }); + return response.status; + }, post.body.number); + expect(removal, "the storage-check post is removed again").toBe(200); }); diff --git a/roles/web-app-fider/files/shell/design-logo-object.sh b/roles/web-app-fider/files/shell/design-logo-object.sh new file mode 100644 index 0000000000..5d919eebe4 --- /dev/null +++ b/roles/web-app-fider/files/shell/design-logo-object.sh @@ -0,0 +1,32 @@ +#!/bin/sh +# Args: action (put|delete), object URL, S3 region. +# Stdin: ":" on the first line; for put, the PNG as base64 after it. +# Prints CHANGED, UNCHANGED, REMOVED or ABSENT. +set -eu +action=$1 +url=$2 +region=$3 +IFS= read -r credentials +config() { + printf 'user = "%s"\naws-sigv4 = "aws:amz:%s:s3"\nurl = "%s"\nsilent\nfail\n' "$credentials" "$region" "$url" +} +if [ "$action" = delete ]; then + if config | curl -K - --head -o /dev/null; then + config | curl -K - -X DELETE -o /dev/null + echo REMOVED + else + echo ABSENT + fi + exit 0 +fi +body=$(mktemp) +trap 'rm -f "$body"' EXIT +base64 -d > "$body" +wanted=$(md5sum < "$body" | cut -d' ' -f1) +current=$(config | curl -K - | md5sum | cut -d' ' -f1) +if [ "$current" = "$wanted" ]; then + echo UNCHANGED + exit 0 +fi +config | curl -K - -X PUT -H 'Content-Type: image/png' --data-binary "@$body" -o /dev/null +echo CHANGED diff --git a/roles/web-app-fider/files/sql/apply-design.sql b/roles/web-app-fider/files/sql/apply-design.sql index 590f8a9059..bd28fb9754 100644 --- a/roles/web-app-fider/files/sql/apply-design.sql +++ b/roles/web-app-fider/files/sql/apply-design.sql @@ -1,4 +1,4 @@ --- Args: title (empty keeps the name), bootstrap_name (the name the role bootstrapped the tenant with), css, logo_key, logo_base64 (empty keeps the logo). +-- Args: title (empty keeps the name), bootstrap_name (the name the role bootstrapped the tenant with), css, logo_key, logo_base64 (non-empty writes the logo into the blobs table), logo_stored (non-empty when the logo is in the active blob store). -- Returns one row: the number of rows that changed. WITH tenant AS ( SELECT id FROM tenants ORDER BY id LIMIT 1 @@ -18,7 +18,7 @@ WITH tenant AS ( THEN %(title)s ELSE tenants.name END AS name, CASE WHEN tenants.custom_css = '' OR position('--infinito-design-carrier' IN tenants.custom_css) > 0 THEN %(css)s ELSE tenants.custom_css END AS custom_css, - CASE WHEN %(logo_base64)s <> '' AND COALESCE(tenants.logo_bkey, '') = '' + CASE WHEN %(logo_stored)s <> '' AND COALESCE(tenants.logo_bkey, '') = '' THEN %(logo_key)s ELSE tenants.logo_bkey END AS logo_bkey FROM tenants JOIN tenant ON tenant.id = tenants.id diff --git a/roles/web-app-fider/files/sql/select-tenant-id.sql b/roles/web-app-fider/files/sql/select-tenant-id.sql new file mode 100644 index 0000000000..35eb9c73b9 --- /dev/null +++ b/roles/web-app-fider/files/sql/select-tenant-id.sql @@ -0,0 +1,2 @@ +-- Returns one row: the id of the tenant the role designs, 0 while no tenant exists. +SELECT COALESCE(MIN(id), 0) FROM tenants; diff --git a/roles/web-app-fider/tasks/02_design.yml b/roles/web-app-fider/tasks/02_design.yml index aa09401cac..7ac9c5f7bd 100644 --- a/roles/web-app-fider/tasks/02_design.yml +++ b/roles/web-app-fider/tasks/02_design.yml @@ -17,6 +17,34 @@ register: fider_design_logo when: FIDER_DESIGN_LOGO_ENABLED | bool +- name: "πŸͺ£ Fider | Logo in the object store" + when: FIDER_OBJSTORE_ENABLED | bool + block: + - name: "πŸͺ£ Fider | Read the tenant id" + database_query: + config: "{{ lookup('database', application_id) }}" + query_file: "{{ role_path }}/files/sql/select-tenant-id.sql" + expect_rows: true + mask_values: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + register: fider_design_tenant + changed_when: false + + - name: "πŸͺ£ Fider | Store or remove the generated logo in the bucket" + ansible.builtin.shell: > + container exec -i {{ FIDER_OBJSTORE_CONTAINER_ADDRESS }} + sh -c {{ lookup('file', 'shell/design-logo-object.sh') | quote }} design-logo-object + {{ 'put' if FIDER_DESIGN_LOGO_ENABLED | bool else 'delete' }} + {{ (lookup('objstore', application_id, 'url') ~ '/' ~ lookup('objstore', application_id, 'bucket') ~ '/tenants/' ~ fider_design_tenant.rows[0][0] ~ '/' ~ + FIDER_DESIGN_LOGO_KEY) | quote }} + {{ lookup('objstore', application_id, 'region') | quote }} + args: + stdin: "{{ lookup('objstore', application_id, 'access_key') }}:{{ lookup('objstore', application_id, 'secret_key') }}\n{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED + | bool else '' }}" + register: fider_design_logo_object + changed_when: fider_design_logo_object.stdout in ['CHANGED', 'REMOVED'] + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: fider_design_tenant.rows[0][0] | int > 0 + - name: "🎨 Fider | Apply title, custom CSS and logo" database_query: config: "{{ lookup('database', application_id) }}" @@ -26,7 +54,8 @@ bootstrap_name: "{{ SOFTWARE_NAME }}" css: "{{ lookup('template', 'custom.css.j2') }}" logo_key: "{{ FIDER_DESIGN_LOGO_KEY }}" - logo_base64: "{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED | bool else '' }}" + logo_base64: "{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED | bool and not (FIDER_OBJSTORE_ENABLED | bool) else '' }}" + logo_stored: "{{ 'true' if FIDER_DESIGN_LOGO_ENABLED | bool else '' }}" expect_rows: true mask_values: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" register: fider_design_apply diff --git a/roles/web-app-fider/vars/main.yml b/roles/web-app-fider/vars/main.yml index 8ef4c8020c..4745a4f016 100644 --- a/roles/web-app-fider/vars/main.yml +++ b/roles/web-app-fider/vars/main.yml @@ -8,9 +8,10 @@ FIDER_OIDC_ENABLED: "{{ lookup('config', application_id, 'services.sso.enabled') FIDER_DESIGN_ENABLED: "{{ lookup('config', application_id, 'services.design.enabled') | bool }}" FIDER_DESIGN_TITLE: "{{ lookup('design', application_id).title if FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).title is string else '' }}" -FIDER_DESIGN_LOGO_ENABLED: "{{ FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).logo is string and not (lookup('objstore', application_id, 'enabled') - | bool) }}" +FIDER_DESIGN_LOGO_ENABLED: "{{ FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).logo is string }}" FIDER_DESIGN_LOGO_KEY: "logos/infinito-design-logo.png" +FIDER_OBJSTORE_ENABLED: "{{ lookup('objstore', application_id, 'enabled') | bool }}" +FIDER_OBJSTORE_CONTAINER_ADDRESS: "{{ lookup('container_address', lookup('objstore', application_id, 'id'), lookup('objstore', application_id, 'engine')) }}" FIDER_TENANT_BOOTSTRAP_BODY: tenantName: "{{ SOFTWARE_NAME }}" From 7069136faae01af51116cbb1513395ceb77663b5 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 21:00:14 +0200 Subject: [PATCH 07/11] docs(requirements): tick requirement 037 for bookwyrm and fider BookWyrm ran its complete spec green on the final state, the SeaweedFS avatar upload included. Fider shows the generated logo with the object store enabled since d912664538. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/requirements/037-corporate-design-overhaul.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/requirements/037-corporate-design-overhaul.md b/docs/requirements/037-corporate-design-overhaul.md index 0493056b0a..5556da18c8 100644 --- a/docs/requirements/037-corporate-design-overhaul.md +++ b/docs/requirements/037-corporate-design-overhaul.md @@ -56,7 +56,7 @@ Two base commits changed many roles after their last run: `6c2de017f8` split ven - [x] web-app-baserow - [ ] web-app-bigbluebutton: title not set through Greenlight (the spec asserts the literal upstream name); the meeting client keeps the upstream design; `vars/main.yml` rewritten after the last full run. - [x] web-app-bluesky -- [ ] web-app-bookwyrm: no green complete run on the final state; the last one failed the SeaweedFS avatar upload test. +- [x] web-app-bookwyrm - [x] web-app-checkmk - [x] web-app-chess - [x] web-app-dashboard @@ -65,7 +65,7 @@ Two base commits changed many roles after their last run: `6c2de017f8` split ven - [x] web-app-erpnext - [x] web-app-espocrm - [x] web-app-fediwall -- [ ] web-app-fider: the logo does not reach Fider while the object store is enabled. +- [x] web-app-fider - [ ] web-app-flowise: `env.j2`, `playwright.env.j2` and `tasks/utils/mcp.yml` changed after the last green run. - [x] web-app-friendica - [x] web-app-funkwhale From d36e2031715db9be87e30624397b1702c9fe7bcd Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 21:25:43 +0200 Subject: [PATCH 08/11] fix(cache): render the applications carrier without the caller's application_id `lookup('tls', ...)` aligns a cross-app domain to the consumer named by `application_id` (utils/tls_common.py `align_domain_to_consumer`): a clearnet-primary consumer gets the clearnet sibling of an onion-primary target. The carrier key does not cover `application_id`, so a park that renders inside a role hands that role's alignment to every later role whose key matches. The constructor and `tasks/utils/setup/role.yml` park at play scope, where `application_id` is undefined. The re-park in `roles/sys-token-store/tasks/write.yml` (67f6c8d7e8) and the nested-load re-park render inside the caller: `svc-prx-openresty` at the preload loader, `web-app-stalwart` or a provider role on the NFS host, the web app itself under `sys-stk-front-base`. In a tor deploy that turned onion targets into clearnet domains for onion-primary roles served from such a park, and baked the caller's group into Nextcloud's sociallogin `groupMapping`. Drop `application_id` from the render's variables and from the templar's for the duration of the render, so every park carries the play-scope result the constructor parks have always carried. Rename the nested re-park task so its log lines do not read as a role boundary. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../files/plugins/lookup/applications.md | 1 + plugins/action/applications_carrier.py | 15 +++++-- tasks/utils/load/app.yml | 2 +- .../action/test_applications_carrier.py | 40 +++++++++++++++++-- 4 files changed, 50 insertions(+), 8 deletions(-) diff --git a/docs/contributing/artefact/files/plugins/lookup/applications.md b/docs/contributing/artefact/files/plugins/lookup/applications.md index c40f2e9d36..1df151b7c9 100644 --- a/docs/contributing/artefact/files/plugins/lookup/applications.md +++ b/docs/contributing/artefact/files/plugins/lookup/applications.md @@ -37,6 +37,7 @@ The constructor stage renders once through the [applications_carrier](../../../. - The stage-wide task sits in `tasks/stages/01_constructor.yml` after the `networks`, `design` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play. - The carrier is parked again after every role in `tasks/utils/setup/role.yml`, after every nested application load in `tasks/utils/load/app.yml`, and after every persisted token in `roles/sys-token-store/tasks/write.yml`. The first two run on every host of the include, also on hosts the loaded role skips. A re-park whose key still matches serves the carried payload without rendering. +- The action renders with `application_id` removed from its variables, so the carried tree holds the play-scope result of every consumer-aligned `lookup('tls', ...)`, whichever role parks it. - The tokens file enters the key as the SHA-256 of its content, memoised per inode, mtime and size, so a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes. - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`. - User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which read the users cache fresh. diff --git a/plugins/action/applications_carrier.py b/plugins/action/applications_carrier.py index bf4b59c1c2..b57393630b 100644 --- a/plugins/action/applications_carrier.py +++ b/plugins/action/applications_carrier.py @@ -70,10 +70,17 @@ class ActionModule(ActionBase): return result revalidate_tokens_file(task_vars) - lookup = lookup_loader.get( - "applications", loader=self._loader, templar=self._templar - ) - carrier = lookup.run([], variables=task_vars, carrier=True)[0] + render_vars = dict(task_vars) + render_vars.pop("application_id", None) + previous = self._templar.available_variables + self._templar.available_variables = render_vars + try: + lookup = lookup_loader.get( + "applications", loader=self._loader, templar=self._templar + ) + carrier = lookup.run([], variables=render_vars, carrier=True)[0] + finally: + self._templar.available_variables = previous result.update( changed=False, diff --git a/tasks/utils/load/app.yml b/tasks/utils/load/app.yml index a4f2265df1..818b4d6278 100644 --- a/tasks/utils/load/app.yml +++ b/tasks/utils/load/app.yml @@ -16,5 +16,5 @@ when: - groups[load_app_id] is not defined or inventory_hostname in groups[load_app_id] -- name: "🧊 Carry the applications SPOT past {{ load_app_id }}" +- name: "🧊 Carry the applications SPOT past nested {{ load_app_id }}" applications_carrier: {} diff --git a/tests/unit/python/plugins/action/test_applications_carrier.py b/tests/unit/python/plugins/action/test_applications_carrier.py index 9f69c1d1d5..4e0c8d213e 100644 --- a/tests/unit/python/plugins/action/test_applications_carrier.py +++ b/tests/unit/python/plugins/action/test_applications_carrier.py @@ -12,21 +12,30 @@ class _FakeTask: self.args = {} if args is None else dict(args) +class _FakeTemplar: + def __init__(self, available_variables=None): + self.available_variables = available_variables + + class _FakeLookup: - def __init__(self, carrier): + def __init__(self, carrier, templar=None): self.carrier = carrier + self.templar = templar self.calls = [] + self.templar_variables = [] def run(self, terms, variables=None, **kwargs): self.calls.append((terms, variables, kwargs)) + if self.templar is not None: + self.templar_variables.append(dict(self.templar.available_variables)) return [self.carrier] -def _make_action(task: _FakeTask) -> ActionModule: +def _make_action(task: _FakeTask, templar=None) -> ActionModule: action = object.__new__(ActionModule) action._task = task action._loader = None - action._templar = None + action._templar = _FakeTemplar() if templar is None else templar return action @@ -56,6 +65,31 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): self.assertEqual(variables, {"applications": {}}) self.assertTrue(kwargs["carrier"]) + def test_renders_without_the_callers_application_id(self): + task_vars = {"applications": {}, "application_id": "web-app-bluesky"} + templar = _FakeTemplar(task_vars) + lookup = _FakeLookup({"key": ["roles", []], "applications": {}}, templar) + action = _make_action(_FakeTask(), templar) + with ( + patch( + "plugins.action.applications_carrier.ActionBase.run", + autospec=True, + return_value={}, + ), + patch( + "plugins.action.applications_carrier.lookup_loader.get", + return_value=lookup, + ), + patch("plugins.action.applications_carrier.revalidate_tokens_file"), + ): + action.run(task_vars=task_vars) + + _, variables, _ = lookup.calls[0] + self.assertEqual(variables, {"applications": {}}) + self.assertEqual(lookup.templar_variables, [{"applications": {}}]) + self.assertIs(templar.available_variables, task_vars) + self.assertEqual(task_vars["application_id"], "web-app-bluesky") + def test_revalidates_the_tokens_file_before_rendering(self): order = [] lookup = _FakeLookup({"key": ["roles", []], "applications": {}}) From 3855b75c5f6c3667043c7f603875c51cf619f3ee Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 22:11:09 +0200 Subject: [PATCH 09/11] fix(sys-ctl-hlth-journalctl): read long and repeated MESSAGE fields Without `--all`, `journalctl --output=json` encodes a field over 4096 bytes as `null`, so a long host error matched nothing and the check passed. A repeated `MESSAGE` field arrives as a JSON array of its values, and `bytes()` on that array raised `TypeError`, which aborted the whole check. Pass `--all`, and tell a byte array (all integers) from an array of field values, decoding each value on its own. Reproduced against the host journal: a 5000-byte error line read back as `"MESSAGE":null` and produced no match; an entry with two `MESSAGE` fields, sent over the native journal socket, read back as `["error first","error second"]` and crashed `host_errors` with `TypeError`. With the fix both are reported. Also clear the two test-lint findings of this PR: `test_comments_valid` rejects the narrative comment above the `journalctl` Popen, so it goes; the stderr-to-file choice stays, because an unread stderr pipe that fills up would block `journalctl` while stdout is still being read. `test_no_uncached_file_reads` flags the read of the stub's argument file, which lives in the test's tempdir and is written per run, so the path-keyed `utils.cache.files.read_text` could serve a stale body; the read opts out with `# nocheck: cache-read tempdir fixture`, as the other stub-driven script tests do. Co-Authored-By: Claude Opus 5.5 (1M context) --- roles/sys-ctl-hlth-journalctl/README.md | 3 ++- .../files/python/script.py | 8 +++++--- .../sys-ctl-hlth-journalctl/files/test_script.py | 16 +++++++++++++++- 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/roles/sys-ctl-hlth-journalctl/README.md b/roles/sys-ctl-hlth-journalctl/README.md index 507b466be1..6f53fb6f81 100644 --- a/roles/sys-ctl-hlth-journalctl/README.md +++ b/roles/sys-ctl-hlth-journalctl/README.md @@ -25,7 +25,8 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional ( ## Features -- Reads the last day of the journal as JSON and drops every entry that carries a `CONTAINER_ID` field. +- Reads the last day of the journal as JSON with every field in full (`--all`) and drops every entry that carries a `CONTAINER_ID` field. +- Joins a repeated `MESSAGE` field into one line before matching. - Matches the remaining messages case-insensitively against `error`. - Prints one `source: message` line per match, naming the systemd unit or, without one, the syslog identifier, so alerts show who logged the error. - Exits non-zero on matches, and also when `journalctl` itself fails, so a broken check never passes silently. diff --git a/roles/sys-ctl-hlth-journalctl/files/python/script.py b/roles/sys-ctl-hlth-journalctl/files/python/script.py index cf0426d3ab..75ddf4483b 100755 --- a/roles/sys-ctl-hlth-journalctl/files/python/script.py +++ b/roles/sys-ctl-hlth-journalctl/files/python/script.py @@ -9,6 +9,7 @@ JOURNALCTL = [ "--since", "1 day ago", "--no-pager", + "--all", "--output=json", "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", ] @@ -18,13 +19,15 @@ def message_text(value): """Return a journal MESSAGE as one line of text. Args: - value: the MESSAGE field; journald encodes non-UTF-8 messages as a list of bytes. + value: the MESSAGE field; journald encodes non-UTF-8 messages as a list of bytes and a repeated field as a list of its values. Returns: The message with line breaks collapsed to spaces. """ - if isinstance(value, list): + if isinstance(value, list) and all(isinstance(item, int) for item in value): value = bytes(value).decode("utf-8", errors="replace") + elif isinstance(value, list): + value = "\n".join(message_text(item) for item in value) return " ".join(str(value).splitlines()) @@ -54,7 +57,6 @@ def host_errors(lines): def main(): print("Checking journalctl for error messages outside container logs...") - # stderr goes to a file, not a pipe: an unread stderr pipe that fills up would block journalctl while stdout is still being read. with tempfile.TemporaryFile( mode="w+", encoding="utf-8", errors="replace" ) as stderr: diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py index 21c17f01c3..8a4f44007f 100644 --- a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py +++ b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py @@ -35,6 +35,10 @@ BINARY_ERROR = { "MESSAGE": list(b"I/O error \xff on disk"), "_SYSTEMD_UNIT": "smartd.service", } +REPEATED_ERROR = { + "MESSAGE": ["unit started", list(b"disk error \xff on sdb")], + "_SYSTEMD_UNIT": "zfs-import.service", +} SYSTEM_INFO = { "MESSAGE": "Started Daily apt download activities.", "_SYSTEMD_UNIT": "init.scope", @@ -71,7 +75,7 @@ class TestHlthJournalctlScript(unittest.TestCase): text=True, check=False, ) - args = (bin_dir / "args").read_text() + args = (bin_dir / "args").read_text() # nocheck: cache-read tempdir fixture return done.returncode, done.stdout, args def test_reads_the_journal_as_json_with_the_source_fields(self): @@ -82,6 +86,16 @@ class TestHlthJournalctlScript(unittest.TestCase): args, ) + def test_reads_every_field_in_full(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--all", args.split()) + + def test_a_repeated_message_field_is_joined_and_reported(self): + code, output, _ = self.run_script(REPEATED_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn("zfs-import.service: unit started disk error", output) + self.assertIn('docker.service: level=error msg="failed to start', output) + def test_an_application_error_from_a_container_is_ignored(self): code, _, _ = self.run_script(APP_ERROR, SYSTEM_INFO) self.assertEqual(code, 0) From 92c1f787713719d37e9e6a1264f043d01ac00138 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 22:56:58 +0200 Subject: [PATCH 10/11] fix(web-app-socialhome): rename the household to the inventory title and back SH_INSTANCE_NAME names the household on the first boot only, and GET /api/instance/config prefers the stored identity row, so a design title set later never reached the household. Upstream renames that row through PATCH /api/admin/instance (admin only, re-broadcast to paired households); the earlier TODO claimed no such route exists, which was wrong. The design tasks now read the current name and converge it with the same semantics as Decidim: household-name.json holds the name the role applied and the one before it. The role renames while the current name is its own, restores the previous name once the title is gone, and leaves a name an administrator chose in the app alone. A fresh install already carries the title from its first boot, so the role records it without a rename; otherwise a later administrator rename would be overwritten on the next deploy. The token, theme and rename tasks moved into tasks/02_design/converge.yml behind one include, because a block may hold at most three tasks. The design spec now checks the instance name and the feed title; the title assertion alone passed with a stale household name, because the design script only replaces the brand suffix. Co-Authored-By: Claude Opus 5.5 (1M context) --- roles/web-app-socialhome/TODO.md | 1 - .../files/playwright/test-design.js | 6 ++ roles/web-app-socialhome/tasks/02_design.yml | 68 +++++++---------- .../tasks/02_design/converge.yml | 73 +++++++++++++++++++ roles/web-app-socialhome/tasks/main.yml | 2 +- roles/web-app-socialhome/vars/main.yml | 3 + 6 files changed, 111 insertions(+), 42 deletions(-) create mode 100644 roles/web-app-socialhome/tasks/02_design/converge.yml diff --git a/roles/web-app-socialhome/TODO.md b/roles/web-app-socialhome/TODO.md index 9a64c909c3..f97a3cd899 100644 --- a/roles/web-app-socialhome/TODO.md +++ b/roles/web-app-socialhome/TODO.md @@ -40,7 +40,6 @@ Open items that only a live run can settle: ## Open from the design pass -- Title override: `SH_INSTANCE_NAME` reaches the home name only on the first boot, because `ensure_instance_identity` (upstream `socialhome/identity_bootstrap.py`) writes `display_name` once and `GET /api/instance/config` prefers that row; an inventory title set later changes the env but not the shown name. No route writes that row at 2026.10.8 (measured: `routes/instance.py` only reads `federation_repo.get_local_identity()`, and no `set`/`update` of the local identity exists in the package), and the row is the identity peers receive. Options: ask upstream for an admin route that renames the household, or accept first-boot only. - Shared sign-in budget: `POST /api/auth/token` allows 5 attempts per 15 minutes per `request.remote`, and the app ignores `X-Forwarded-For`, so every browser behind the front proxy and the deploy's own token request share one bucket (measured: all logins in `/api/admin/auth-audit` come from `192.168.75.1`). At 2026.10.11 the budget is the module constant `AUTH_TOKEN_RATE_LIMIT` in `socialhome/routes/users.py` with no env or TOML key, and `trusted_proxies` exists for the GFS only (`socialhome/global_server/config.py`); the design spec keeps one administrator token in the Playwright staging volume. Options: let the proxy reach the app with the client address (PROXY protocol or upstream trusted-proxy support), or ask upstream to honour a trusted forwarded header. - `tasks/01_setup.yml` comment is stale: at 2026.10.8 the headless seed marks setup complete itself (upstream `platform/standalone/adapter.py`), so the task always answers 409 (measured: `ok`, never `changed`). Options: drop the task or rewrite the comment. - `meta/csp.yml` keeps `script-src-elem unsafe-inline` for an inline theme script the 2026.10.8 shell no longer has (it loads `assets/theme-boot.js`; measured: the served `/feed` carries no inline script of its own). Options: drop the flag after checking `hlth-csp`; the injected design script then needs its CSP hash, which `sys-front-inj-design` adds. diff --git a/roles/web-app-socialhome/files/playwright/test-design.js b/roles/web-app-socialhome/files/playwright/test-design.js index 6116b6edad..be0a2b4a34 100644 --- a/roles/web-app-socialhome/files/playwright/test-design.js +++ b/roles/web-app-socialhome/files/playwright/test-design.js @@ -569,6 +569,12 @@ exports.register = function () { } await expect(page.locator("link[rel~='icon']").first()).toHaveAttribute("href", faviconUrl); } + if (title) { + const config = await apiGetOnion(page.request, `${base}/api/instance/config`); + expect((await config.json()).instance_name, "the household carries the configured title").toBe(title); + await visit(page, `${base}/feed`); + await expect(page.locator(TOPBAR_TITLE), "the feed is named after the household").toHaveText(title); + } }); test("design: the sign-in page shows the generated logo", async ({ page }) => { diff --git a/roles/web-app-socialhome/tasks/02_design.yml b/roles/web-app-socialhome/tasks/02_design.yml index 33a79c9f01..c133fae330 100644 --- a/roles/web-app-socialhome/tasks/02_design.yml +++ b/roles/web-app-socialhome/tasks/02_design.yml @@ -5,48 +5,20 @@ checksum_algorithm: sha1 register: socialhome_theme_marker -- name: "🎨 Converge the household primary colour" - when: SOCIALHOME_DESIGN_ENABLED | bool or socialhome_theme_marker.stat.exists - block: - - name: "πŸ”‘ Request an administrator token for the household theme" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/auth/token" - method: POST - body_format: json - body: - username: "{{ SOCIALHOME_ADMIN_USERNAME }}" - password: "{{ SOCIALHOME_ADMIN_PASSWORD }}" - status_code: 200 - register: socialhome_theme_token - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "πŸ“₯ Look up the household name the role wrote last" + ansible.builtin.stat: + path: "{{ SOCIALHOME_NAME_STATE_FILE }}" + register: socialhome_name_marker - - name: "🎨 Read the household theme" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" - method: GET - headers: - Authorization: "Bearer {{ socialhome_theme_token.json.token }}" - status_code: 200 - register: socialhome_theme - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "πŸ“₯ Read the household name the role wrote last" + ansible.builtin.slurp: + src: "{{ SOCIALHOME_NAME_STATE_FILE }}" + register: socialhome_name_state_file + when: socialhome_name_marker.stat.exists - - name: "🎨 Write the household primary colour" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" - method: PUT - body_format: json - body: - primary_color: "{{ SOCIALHOME_THEME_PRIMARY }}" - headers: - Authorization: "Bearer {{ socialhome_theme_token.json.token }}" - status_code: 200 - register: socialhome_theme_written - changed_when: true - when: - - socialhome_theme.json.primary_color | lower != SOCIALHOME_THEME_PRIMARY - - socialhome_theme.json.primary_color | lower == SOCIALHOME_BRAND_PRIMARY or (socialhome_theme_marker.stat.exists and - socialhome_theme_marker.stat.checksum == (socialhome_theme.json.primary_color | lower | hash('sha1'))) - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "🎨 Converge the household primary colour and name" + include_tasks: 02_design/converge.yml + when: SOCIALHOME_DESIGN_ENABLED | bool or socialhome_theme_marker.stat.exists or socialhome_name_marker.stat.exists - name: "🏷️ Mark the household primary colour as the role's" stack_host_copy: @@ -57,8 +29,24 @@ - SOCIALHOME_DESIGN_ENABLED | bool - socialhome_theme_written is changed +- name: "🏷️ Mark the household name as the role's" + stack_host_copy: + content: "{{ {'applied': SOCIALHOME_DESIGN_TITLE, 'previous': (socialhome_name_state.previous if socialhome_name_state | length > 0 else socialhome_name_current)} + | to_json }}" + dest: "{{ SOCIALHOME_NAME_STATE_FILE }}" + mode: "0644" + when: + - SOCIALHOME_DESIGN_TITLE | length > 0 + - socialhome_name_written is changed or (socialhome_name_state | length == 0 and socialhome_name_current == SOCIALHOME_DESIGN_TITLE) + - name: "🧹 Drop the household theme marker" ansible.builtin.file: path: "{{ SOCIALHOME_THEME_MARKER_FILE }}" state: absent when: not (SOCIALHOME_DESIGN_ENABLED | bool) + +- name: "🧹 Drop the household name marker" + ansible.builtin.file: + path: "{{ SOCIALHOME_NAME_STATE_FILE }}" + state: absent + when: SOCIALHOME_DESIGN_TITLE | length == 0 diff --git a/roles/web-app-socialhome/tasks/02_design/converge.yml b/roles/web-app-socialhome/tasks/02_design/converge.yml new file mode 100644 index 0000000000..ac72fc6129 --- /dev/null +++ b/roles/web-app-socialhome/tasks/02_design/converge.yml @@ -0,0 +1,73 @@ +--- +- name: "πŸ”‘ Request an administrator token for the household theme" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/auth/token" + method: POST + body_format: json + body: + username: "{{ SOCIALHOME_ADMIN_USERNAME }}" + password: "{{ SOCIALHOME_ADMIN_PASSWORD }}" + status_code: 200 + register: socialhome_theme_token + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🎨 Read the household theme" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" + method: GET + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + register: socialhome_theme + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🎨 Write the household primary colour" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" + method: PUT + body_format: json + body: + primary_color: "{{ SOCIALHOME_THEME_PRIMARY }}" + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + register: socialhome_theme_written + changed_when: true + when: + - socialhome_theme.json.primary_color | lower != SOCIALHOME_THEME_PRIMARY + - socialhome_theme.json.primary_color | lower == SOCIALHOME_BRAND_PRIMARY or (socialhome_theme_marker.stat.exists and socialhome_theme_marker.stat.checksum + == (socialhome_theme.json.primary_color | lower | hash('sha1'))) + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🏷️ Read the household name" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/instance/config" + method: GET + status_code: 200 + register: socialhome_instance_config + +- name: "🏷️ Hold the household name and the name the role wrote last" + ansible.builtin.set_fact: + socialhome_name_current: "{{ socialhome_instance_config.json.instance_name }}" + socialhome_name_state: "{{ (socialhome_name_state_file.content | b64decode | from_json) if socialhome_name_marker.stat.exists else {} }}" + +- name: "🏷️ Rename the household" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/admin/instance" + method: PATCH + body_format: json + body: + display_name: "{{ socialhome_name_target }}" + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + vars: + socialhome_name_ours: "{{ socialhome_name_state | length > 0 and socialhome_name_current == socialhome_name_state.applied }}" + socialhome_name_target: >- + {{ (socialhome_name_state.previous if socialhome_name_ours | bool else '') if SOCIALHOME_DESIGN_TITLE | length == 0 + else ('' if (socialhome_name_state | length > 0 and not (socialhome_name_ours | bool)) or socialhome_name_current == SOCIALHOME_DESIGN_TITLE + else SOCIALHOME_DESIGN_TITLE) }} + register: socialhome_name_written + changed_when: true + when: socialhome_name_target | length > 0 + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" diff --git a/roles/web-app-socialhome/tasks/main.yml b/roles/web-app-socialhome/tasks/main.yml index 17fb8f576a..c13e243e8f 100644 --- a/roles/web-app-socialhome/tasks/main.yml +++ b/roles/web-app-socialhome/tasks/main.yml @@ -22,5 +22,5 @@ - name: "πŸ” Complete the standalone first-boot setup for '{{ application_id }}'" include_tasks: 01_setup.yml - - name: "🎨 Converge the household theme for '{{ application_id }}'" + - name: "🎨 Converge the household theme and name for '{{ application_id }}'" include_tasks: 02_design.yml diff --git a/roles/web-app-socialhome/vars/main.yml b/roles/web-app-socialhome/vars/main.yml index f94df8960c..7caa51932f 100644 --- a/roles/web-app-socialhome/vars/main.yml +++ b/roles/web-app-socialhome/vars/main.yml @@ -16,6 +16,9 @@ SOCIALHOME_CONFIG_HOST_FILE: "{{ [lookup('container', application_id, 'directori SOCIALHOME_MAX_STORAGE_BYTES: 4294967296 SOCIALHOME_DESIGN_ENABLED: "{{ lookup('config', application_id, 'services.design.enabled') | bool }}" +SOCIALHOME_DESIGN_TITLE: "{{ lookup('design', application_id).title if SOCIALHOME_DESIGN_ENABLED | bool and lookup('design', application_id).title is string else + '' }}" +SOCIALHOME_NAME_STATE_FILE: "{{ [lookup('container', application_id, 'directories.volumes'), 'household-name.json'] | path_join }}" SOCIALHOME_BRAND_PRIMARY: "#d2542a" SOCIALHOME_THEME_PRIMARY: >- {{ (lookup('design_palette', lookup('config', 'web-svc-design', 'services.design.colors.base')).light['--design-primary'] | lower) From cc154a8663f2b69cebe8e76129d87d39a51cd041 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 16 Oct 2026 22:58:18 +0200 Subject: [PATCH 11/11] docs(requirements): tick the title override item of requirement 037 Socialhome renames an existing household to an inventory title and back since 92c1f78771, so both roles of the item honour the override. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/requirements/037-corporate-design-overhaul.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/037-corporate-design-overhaul.md b/docs/requirements/037-corporate-design-overhaul.md index 5556da18c8..003583f7c7 100644 --- a/docs/requirements/037-corporate-design-overhaul.md +++ b/docs/requirements/037-corporate-design-overhaul.md @@ -42,7 +42,7 @@ These choices are settled at requirement creation time and bound the implementat - [x] A unit test proves the logo and title precedence: global value, per-role override, and `false`/`0` disabling. - [x] No `roles/*/meta/services.yml` declares a site title as `title`, `titel`, `sitename`, `site_name` or `site_titel` on its primary entity; every site title lives in the role's `design` entry and is read through `lookup('design', application_id)`. - [x] `web-app-joomla` and `web-app-pretix` carry no hardcoded site title. -- [ ] `web-app-decidim` and `web-app-socialhome` honour an inventory title override. Decidim renames an existing organization to an inventory title and back. Open: Socialhome takes the title on its first boot only (`roles/web-app-socialhome/TODO.md`). +- [x] `web-app-decidim` and `web-app-socialhome` honour an inventory title override. Decidim renames an existing organization and Socialhome an existing household to an inventory title and back. - [x] The base gallery on `web-app-dashboard`, `web-app-gitea` and `web-app-nextcloud` is approved by the operator. - [ ] `feature/designer` reaches `main` through the draft PR of `feature/store-i18n` linked below, which merges it round by round. Open: the final designer state is not yet pushed into that PR.