diff --git a/docs/contributing/artefact/files/plugins/lookup/applications.md b/docs/contributing/artefact/files/plugins/lookup/applications.md index 67061c3713..46577fc3d7 100644 --- a/docs/contributing/artefact/files/plugins/lookup/applications.md +++ b/docs/contributing/artefact/files/plugins/lookup/applications.md @@ -36,7 +36,8 @@ Ansible forks a fresh worker process for every task. The in-process cache inside The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY`, `SYSTEM_EMAIL_DOMAIN` or `group_names`, or a changed path or content of the tokens file). - The stage-wide task sits in `tasks/stages/01_constructor.yml` after the `networks` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play. -- The carrier is parked again after every role in `tasks/utils/setup/role.yml` and after every persisted token in `roles/sys-token-store/tasks/write.yml`. A re-park whose key still matches serves the carried payload without rendering. +- The carrier is parked again after every role in `tasks/utils/setup/role.yml`, after every nested application load in `tasks/utils/load/app.yml`, and after every persisted token in `roles/sys-token-store/tasks/write.yml`. The first two run on every host of the include, also on hosts the loaded role skips. A re-park whose key still matches serves the carried payload without rendering. +- The action renders with `application_id` removed from its variables, so the carried tree holds the play-scope result of every consumer-aligned `lookup('tls', ...)`, whichever role parks it. - The tokens file enters the key as the SHA-256 of its content, memoised per inode, mtime and size, so a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes. - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`. - User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which read the users cache fresh. diff --git a/plugins/action/applications_carrier.py b/plugins/action/applications_carrier.py index 93c479ff3d..b57393630b 100644 --- a/plugins/action/applications_carrier.py +++ b/plugins/action/applications_carrier.py @@ -21,7 +21,8 @@ longer match is rejected and the reader renders afresh:: applications_carrier: clear: true -``tasks/utils/setup/role.yml`` parks one after every role and +``tasks/utils/setup/role.yml`` parks one after every role, +``tasks/utils/load/app.yml`` one after every nested application load and ``roles/sys-token-store/tasks/write.yml`` one after every persisted token; where the key still matches they serve the carried payload without rendering. @@ -69,10 +70,17 @@ class ActionModule(ActionBase): return result revalidate_tokens_file(task_vars) - lookup = lookup_loader.get( - "applications", loader=self._loader, templar=self._templar - ) - carrier = lookup.run([], variables=task_vars, carrier=True)[0] + render_vars = dict(task_vars) + render_vars.pop("application_id", None) + previous = self._templar.available_variables + self._templar.available_variables = render_vars + try: + lookup = lookup_loader.get( + "applications", loader=self._loader, templar=self._templar + ) + carrier = lookup.run([], variables=render_vars, carrier=True)[0] + finally: + self._templar.available_variables = previous result.update( changed=False, diff --git a/roles/sys-ctl-hlth-journalctl/README.md b/roles/sys-ctl-hlth-journalctl/README.md index c60c77d27d..6f53fb6f81 100644 --- a/roles/sys-ctl-hlth-journalctl/README.md +++ b/roles/sys-ctl-hlth-journalctl/README.md @@ -2,11 +2,13 @@ ## Description -Scans `journalctl` over the last day for “error” entries and alerts if any are found. +Scans `journalctl` over the last day for “error” entries outside container logs and alerts if any are found. ## Overview This role searches the systemd journal for errors over the past day and alerts if any are found. +Container output is skipped, because every compose service logs to journald and application errors would otherwise trigger the host check. +Docker daemon and system errors are still reported. ## Cosmos @@ -23,8 +25,11 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional ( ## Features -- Runs `journalctl --since '1 day ago' | grep -i error`. -- Exits non-zero on matches. +- Reads the last day of the journal as JSON with every field in full (`--all`) and drops every entry that carries a `CONTAINER_ID` field. +- Joins a repeated `MESSAGE` field into one line before matching. +- Matches the remaining messages case-insensitively against `error`. +- Prints one `source: message` line per match, naming the systemd unit or, without one, the syslog identifier, so alerts show who logged the error. +- Exits non-zero on matches, and also when `journalctl` itself fails, so a broken check never passes silently. - Scheduled via systemd timer. - Alerts via `sys-ctl-alm-compose` on detection. diff --git a/roles/sys-ctl-hlth-journalctl/files/python/script.py b/roles/sys-ctl-hlth-journalctl/files/python/script.py new file mode 100755 index 0000000000..75ddf4483b --- /dev/null +++ b/roles/sys-ctl-hlth-journalctl/files/python/script.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +import json +import subprocess +import sys +import tempfile + +JOURNALCTL = [ + "journalctl", + "--since", + "1 day ago", + "--no-pager", + "--all", + "--output=json", + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", +] + + +def message_text(value): + """Return a journal MESSAGE as one line of text. + + Args: + value: the MESSAGE field; journald encodes non-UTF-8 messages as a list of bytes and a repeated field as a list of its values. + + Returns: + The message with line breaks collapsed to spaces. + """ + if isinstance(value, list) and all(isinstance(item, int) for item in value): + value = bytes(value).decode("utf-8", errors="replace") + elif isinstance(value, list): + value = "\n".join(message_text(item) for item in value) + return " ".join(str(value).splitlines()) + + +def host_errors(lines): + """Yield `source: message` for every non-container entry that mentions an error. + + Args: + lines: iterable of journalctl JSON lines, one entry per line, consumed lazily. + + Returns: + Iterator of readable error lines. + """ + for line in lines: + if not line.strip(): + continue + entry = json.loads(line) + if "CONTAINER_ID" in entry: + continue + message = message_text(entry.get("MESSAGE", "")) + if "error" not in message.lower(): + continue + source = ( + entry.get("_SYSTEMD_UNIT") or entry.get("SYSLOG_IDENTIFIER") or "unknown" + ) + yield f"{source}: {message}" + + +def main(): + print("Checking journalctl for error messages outside container logs...") + with tempfile.TemporaryFile( + mode="w+", encoding="utf-8", errors="replace" + ) as stderr: + with subprocess.Popen( + JOURNALCTL, + stdout=subprocess.PIPE, + stderr=stderr, + encoding="utf-8", + errors="replace", + ) as journal: + errors = list(host_errors(journal.stdout)) + if journal.returncode != 0: + stderr.seek(0) + print( + f"journalctl failed with exit code {journal.returncode}: {stderr.read().strip()}" + ) + return 2 + if errors: + print("Some errors were found:") + print("\n".join(errors)) + return 1 + print("No errors found in the journal.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh b/roles/sys-ctl-hlth-journalctl/files/shell/script.sh deleted file mode 100644 index 4903b597d1..0000000000 --- a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -echo "Checking journalctl for error messages..." -journalctl_errors="$(journalctl --since '1 day ago' --no-pager | grep -i 'error')" -if [ -n "$journalctl_errors" ] -then - echo "Some errors where found: $journalctl_errors" - exit 1 -fi -echo "All docker containers are healthy." -exit 0 diff --git a/roles/web-app-suitecrm/README.md b/roles/web-app-suitecrm/README.md index c1cbcd6dd8..aff1131267 100644 --- a/roles/web-app-suitecrm/README.md +++ b/roles/web-app-suitecrm/README.md @@ -143,6 +143,14 @@ docker run --rm -it \ the Tor SOCKS proxy whenever the issuer is an onion, because a `.onion` issuer is unresolvable directly. `/auth` stays reachable as the local break-glass login. +- **Auto-created accounts** reach `User::save()` through SuiteCRM's own `CreateUser` + while the request is still anonymous. Since 8.10.2 that save answers + `RuntimeException: Not authorized` and forces `status = Inactive`, so + `files/patch/external-auth-user-create.patch` makes the stored system user the current + user around the save and restores the caller's afterwards. + [`files/Dockerfile`](./files/Dockerfile) applies it with `patch -F0` on the unpacked + release; the image build fails once the hunk stops matching. + ## Persona contract opt-outs [`templates/env.j2`](./templates/env.j2) seeds only the administrator into SuiteCRM (`SUITECRM_ADMIN_USERNAME` / `SUITECRM_ADMIN_PASSWORD`); `biber` reaches the app through SuiteCRM's own SAML login against Keycloak, where `SAML_AUTO_CREATE` mints the account on first assertion. In the `services.sso.enabled: false` matrix variants there is no SAML entry point and `biber` has no native account, so [`templates/playwright.env.j2`](./templates/playwright.env.j2) renders `PERSONA_BIBER_BLOCKED=true` for the generic persona journeys (the LDAP spec still signs `biber` in from the directory). The `administrator` persona keeps its native login via `ADMIN_NATIVE_PASSWORD` and runs in every variant. diff --git a/roles/web-app-suitecrm/files/Dockerfile b/roles/web-app-suitecrm/files/Dockerfile index 08c8f95e71..a2a6a6ec0e 100644 --- a/roles/web-app-suitecrm/files/Dockerfile +++ b/roles/web-app-suitecrm/files/Dockerfile @@ -19,6 +19,7 @@ RUN rm -rf /var/lib/apt/lists/* \ libonig-dev \ libxml2-dev \ libldap2-dev \ + patch \ && docker-php-ext-configure ldap --with-libdir=lib/x86_64-linux-gnu \ && docker-php-ext-install \ pdo \ @@ -56,6 +57,8 @@ FROM base AS builder ARG SUITECRM_APP_VERSION ARG SUITECRM_APP_DIR ARG SUITECRM_LDAP_EXTENSION_DIR +ARG SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL +ARG SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER WORKDIR ${SUITECRM_APP_DIR} @@ -69,6 +72,11 @@ RUN set -eux; \ rm /tmp/suitecrm.tar.gz; \ chmod +x bin/console +COPY ${SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL} ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER} +RUN set -eux; \ + patch -p1 -F0 -i ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER}; \ + rm ${SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER} + RUN set -eux; \ composer install \ --prefer-dist \ diff --git a/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch new file mode 100644 index 0000000000..6f7275b53d --- /dev/null +++ b/roles/web-app-suitecrm/files/patch/external-auth-user-create.patch @@ -0,0 +1,69 @@ +SuiteCRM 8.10.2 tightened User::hasSaveAccess() from `return true` to +`return is_admin($current_user)` for a record without an id, and +restrictAdminOnlyFields() now forces `status = Inactive` on such a record +(release commit 6c7ce002a). CreateUser::create() runs while the browser is +still anonymous, because it is the LDAP and SAML auto-create path +(AppLdapUserProvider::createUser, AppSamlUserFactory::createUser, both via +UserHandler::createExternalAuthUser), so every first login of a directory +account ends in `RuntimeException: Not authorized`, HTTP 500 on POST /login +and no session after a SAML assertion. v8.10.1 and earlier exempted a new +record from both guards; upstream has no release and no hotfix carrying a fix. + +\CreateUser is reachable only from that provisioning stack and from the +installer, so the admin context is raised here instead of relaxing the guards, +which keeps them closed for every request-driven save. + +The raised context is the stored system user, never a fresh in-memory bean +carrying `is_admin = 1`. User::verify_data() compares `$this->id == +$current_user->id`, which is `'' == ''` between two unsaved beans, so a +deployment holding a single administrator takes the last-admin branch for the +very account it is creating. That branch reads the legacy `$mod_strings`, which +this request never loads, so an LDAP login ends in `ErrorException: Trying to +access array offset on value of type null` at modules/Users/User.php:1678 and +HTTP 500 on POST /login, while a SAML assertion redirects to +`index.php?action=Error&module=Users` and leaves no session. + +`$current_user` is assigned before `getSystemUser()` runs, not from its return +value. User::retrieve() calls loadPreferences() after it has filled the bean, +and both UserPreference::loadPreferences() and reloadPreferences() dereference +`$current_user->id` unguarded, so retrieving while the caller's anonymous +`$current_user` is still in place would raise `Attempt to read property "id" on +null` and answer HTTP 500 again. Assigning first makes the global the very bean +being filled, so those reads see an object throughout and the only session key +written is the system user's own. + +Generated with `diff -u` against the v8.10.2 tarball and applied with `-F0`, so +the build fails instead of letting GNU patch's default fuzz factor land the +hunk somewhere upstream moved it to. + +--- a/public/legacy/include/portability/User/CreateUser.php ++++ b/public/legacy/include/portability/User/CreateUser.php +@@ -68,6 +68,8 @@ + */ + public function create(string $name, array $fields): ?SugarBean + { ++ global $current_user; ++ + $user = BeanFactory::newBean('Users'); + $user->user_name = $name; + +@@ -77,7 +79,18 @@ + $user->$key = $value; + } + +- $user->save(); ++ $callerUser = $current_user; ++ ++ if (!is_admin($current_user)) { ++ $current_user = BeanFactory::newBean('Users'); ++ $current_user->getSystemUser(); ++ } ++ ++ try { ++ $user->save(); ++ } finally { ++ $current_user = $callerUser; ++ } + + if (empty($user)) { + return null; diff --git a/roles/web-app-suitecrm/meta/services.yml b/roles/web-app-suitecrm/meta/services.yml index 40902296c2..3979629734 100644 --- a/roles/web-app-suitecrm/meta/services.yml +++ b/roles/web-app-suitecrm/meta/services.yml @@ -59,7 +59,7 @@ suitecrm: # nocheck: docker-version version: 8.2-apache custom: true - app_version: 8.9.1 + app_version: 8.10.2 phpredis_version: 6.3.0 update: - key: app_version diff --git a/roles/web-app-suitecrm/tasks/01_manager_ops.yml b/roles/web-app-suitecrm/tasks/01_manager_ops.yml index 373126537c..f0a6bc481a 100644 --- a/roles/web-app-suitecrm/tasks/01_manager_ops.yml +++ b/roles/web-app-suitecrm/tasks/01_manager_ops.yml @@ -30,6 +30,14 @@ - compose-up - compose-build +- name: "🚀 Deploy '{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS }}'" + copy: + src: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_SRC }}" + dest: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS }}" + notify: + - compose-up + - compose-build + - name: "🐳 Docker Compose Up for '{{ application_id }}'" meta: flush_handlers diff --git a/roles/web-app-suitecrm/templates/compose.yml.j2 b/roles/web-app-suitecrm/templates/compose.yml.j2 index a4a9f433e8..2735bcb41b 100644 --- a/roles/web-app-suitecrm/templates/compose.yml.j2 +++ b/roles/web-app-suitecrm/templates/compose.yml.j2 @@ -9,6 +9,8 @@ SUITECRM_PHPREDIS_VERSION: "{{ SUITECRM_PHPREDIS_VERSION }}" SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL: "{{ SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL }}" SUITECRM_ENTRYPOINT_SCRIPT_DOCKER: "{{ SUITECRM_ENTRYPOINT_SCRIPT_DOCKER }}" + SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL }}" + SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER: "{{ SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER }}" SUITECRM_LDAP_EXTENSION_DIR: "{{ SUITECRM_LDAP_EXTENSION_DIR }}" SUITECRM_APP_DIR: "{{ SUITECRM_APP_DIR }}" SUITECRM_PHP_MEMORY_LIMIT: "{{ SUITECRM_PHP_MEMORY_LIMIT }}" diff --git a/roles/web-app-suitecrm/vars/main.yml b/roles/web-app-suitecrm/vars/main.yml index 2f1220b2d7..05b48ed2d5 100644 --- a/roles/web-app-suitecrm/vars/main.yml +++ b/roles/web-app-suitecrm/vars/main.yml @@ -41,6 +41,12 @@ SUITECRM_ENTRYPOINT_SCRIPT_HOST_ABS: "{{ [ lookup('container', application_id, ' SUITECRM_ENTRYPOINT_SCRIPT_HOST_REL: "volumes/{{ SUITECRM_ENTRYPOINT_SCRIPT_FILE }}" SUITECRM_ENTRYPOINT_SCRIPT_DOCKER: "{{ [ '/usr/local/bin/', SUITECRM_ENTRYPOINT_SCRIPT_FILE ] | path_join }}" +SUITECRM_EXTERNAL_AUTH_PATCH_FILE: "external-auth-user-create.patch" +SUITECRM_EXTERNAL_AUTH_PATCH_SRC: "patch/{{ SUITECRM_EXTERNAL_AUTH_PATCH_FILE }}" +SUITECRM_EXTERNAL_AUTH_PATCH_HOST_ABS: "{{ [ lookup('container', application_id, 'directories.volumes'), SUITECRM_EXTERNAL_AUTH_PATCH_FILE ] | path_join }}" +SUITECRM_EXTERNAL_AUTH_PATCH_HOST_REL: "volumes/{{ SUITECRM_EXTERNAL_AUTH_PATCH_FILE }}" +SUITECRM_EXTERNAL_AUTH_PATCH_DOCKER: "{{ [ '/tmp/', SUITECRM_EXTERNAL_AUTH_PATCH_FILE ] | path_join }}" + SUITECRM_DESIGN_ENABLED: "{{ lookup('config', application_id, 'services.design.enabled') | bool }}" SUITECRM_DESIGN_LOGO_ENABLED: "{{ SUITECRM_DESIGN_ENABLED | bool and lookup('design', application_id).logo is string }}" SUITECRM_DESIGN_TITLE: "{{ lookup('design', application_id).title if SUITECRM_DESIGN_ENABLED | bool and lookup('design', application_id).title is string else '' }}" diff --git a/tasks/utils/load/app.yml b/tasks/utils/load/app.yml index abf1e21cb4..818b4d6278 100644 --- a/tasks/utils/load/app.yml +++ b/tasks/utils/load/app.yml @@ -15,3 +15,6 @@ domain: "{{ lookup('domain', load_app_id) }}" when: - groups[load_app_id] is not defined or inventory_hostname in groups[load_app_id] + +- name: "🧊 Carry the applications SPOT past nested {{ load_app_id }}" + applications_carrier: {} diff --git a/tests/unit/python/plugins/action/test_applications_carrier.py b/tests/unit/python/plugins/action/test_applications_carrier.py index 9f69c1d1d5..4e0c8d213e 100644 --- a/tests/unit/python/plugins/action/test_applications_carrier.py +++ b/tests/unit/python/plugins/action/test_applications_carrier.py @@ -12,21 +12,30 @@ class _FakeTask: self.args = {} if args is None else dict(args) +class _FakeTemplar: + def __init__(self, available_variables=None): + self.available_variables = available_variables + + class _FakeLookup: - def __init__(self, carrier): + def __init__(self, carrier, templar=None): self.carrier = carrier + self.templar = templar self.calls = [] + self.templar_variables = [] def run(self, terms, variables=None, **kwargs): self.calls.append((terms, variables, kwargs)) + if self.templar is not None: + self.templar_variables.append(dict(self.templar.available_variables)) return [self.carrier] -def _make_action(task: _FakeTask) -> ActionModule: +def _make_action(task: _FakeTask, templar=None) -> ActionModule: action = object.__new__(ActionModule) action._task = task action._loader = None - action._templar = None + action._templar = _FakeTemplar() if templar is None else templar return action @@ -56,6 +65,31 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): self.assertEqual(variables, {"applications": {}}) self.assertTrue(kwargs["carrier"]) + def test_renders_without_the_callers_application_id(self): + task_vars = {"applications": {}, "application_id": "web-app-bluesky"} + templar = _FakeTemplar(task_vars) + lookup = _FakeLookup({"key": ["roles", []], "applications": {}}, templar) + action = _make_action(_FakeTask(), templar) + with ( + patch( + "plugins.action.applications_carrier.ActionBase.run", + autospec=True, + return_value={}, + ), + patch( + "plugins.action.applications_carrier.lookup_loader.get", + return_value=lookup, + ), + patch("plugins.action.applications_carrier.revalidate_tokens_file"), + ): + action.run(task_vars=task_vars) + + _, variables, _ = lookup.calls[0] + self.assertEqual(variables, {"applications": {}}) + self.assertEqual(lookup.templar_variables, [{"applications": {}}]) + self.assertIs(templar.available_variables, task_vars) + self.assertEqual(task_vars["application_id"], "web-app-bluesky") + def test_revalidates_the_tokens_file_before_rendering(self): order = [] lookup = _FakeLookup({"key": ["roles", []], "applications": {}}) diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py new file mode 100644 index 0000000000..8a4f44007f --- /dev/null +++ b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py @@ -0,0 +1,147 @@ +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from utils import PROJECT_ROOT + +SCRIPT = PROJECT_ROOT / "roles/sys-ctl-hlth-journalctl/files/python/script.py" + +JOURNALCTL_STUB = """#!/usr/bin/env bash +printf '%s\\n' "$*" >"${STUB_DIR}/args" +cat "${STUB_DIR}/journal" +if [ "${STUB_RC}" -ne 0 ]; then +\techo "journalctl: unrecognized option '--output-fields'" >&2 +fi +exit "${STUB_RC}" +""" + +APP_ERROR = { + "MESSAGE": "ERROR: pixelfed queue failed", + "CONTAINER_ID": "0123456789ab", + "SYSLOG_IDENTIFIER": "pixelfed", + "_SYSTEMD_UNIT": "docker.service", +} +DAEMON_ERROR = { + "MESSAGE": 'level=error msg="failed to start container"', + "SYSLOG_IDENTIFIER": "dockerd", + "_SYSTEMD_UNIT": "docker.service", +} +KERNEL_ERROR = {"MESSAGE": "EXT4-fs error (device sda1)", "SYSLOG_IDENTIFIER": "kernel"} +BINARY_ERROR = { + "MESSAGE": list(b"I/O error \xff on disk"), + "_SYSTEMD_UNIT": "smartd.service", +} +REPEATED_ERROR = { + "MESSAGE": ["unit started", list(b"disk error \xff on sdb")], + "_SYSTEMD_UNIT": "zfs-import.service", +} +SYSTEM_INFO = { + "MESSAGE": "Started Daily apt download activities.", + "_SYSTEMD_UNIT": "init.scope", +} + + +class TestHlthJournalctlScript(unittest.TestCase): + def run_script(self, *entries, rc=0): + """Run the health script against a stubbed journalctl. + + Args: + entries: journal entries the stub prints, one JSON object per line. + rc: exit code of the stubbed journalctl. + + Returns: + Tuple of the exit code, the script output, and the journalctl arguments. + """ + with tempfile.TemporaryDirectory() as tmp: + bin_dir = Path(tmp) + stub = bin_dir / "journalctl" + stub.write_text(JOURNALCTL_STUB) + stub.chmod(0o755) + (bin_dir / "journal").write_text( + "".join(json.dumps(entry) + "\n" for entry in entries) + ) + env = dict(os.environ) + env["PATH"] = f"{bin_dir}{os.pathsep}{env['PATH']}" + env["STUB_DIR"] = tmp + env["STUB_RC"] = str(rc) + done = subprocess.run( + [sys.executable, str(SCRIPT)], + env=env, + capture_output=True, + text=True, + check=False, + ) + args = (bin_dir / "args").read_text() # nocheck: cache-read tempdir fixture + return done.returncode, done.stdout, args + + def test_reads_the_journal_as_json_with_the_source_fields(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--output=json", args) + self.assertIn( + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", + args, + ) + + def test_reads_every_field_in_full(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--all", args.split()) + + def test_a_repeated_message_field_is_joined_and_reported(self): + code, output, _ = self.run_script(REPEATED_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn("zfs-import.service: unit started disk error", output) + self.assertIn('docker.service: level=error msg="failed to start', output) + + def test_an_application_error_from_a_container_is_ignored(self): + code, _, _ = self.run_script(APP_ERROR, SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_docker_daemon_error_is_reported_with_its_unit(self): + code, output, _ = self.run_script(APP_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn( + 'docker.service: level=error msg="failed to start container"', output + ) + self.assertNotIn("pixelfed queue failed", output) + + def test_an_entry_without_unit_is_named_by_its_identifier(self): + code, output, _ = self.run_script(KERNEL_ERROR) + self.assertEqual(code, 1) + self.assertIn("kernel: EXT4-fs error (device sda1)", output) + + def test_a_non_utf8_message_is_decoded_and_reported(self): + code, output, _ = self.run_script(BINARY_ERROR) + self.assertEqual(code, 1) + self.assertIn("smartd.service: I/O error", output) + + def test_many_container_errors_are_streamed_past_and_not_reported(self): + container_lines = [ + {**APP_ERROR, "MESSAGE": f"ERROR: pixelfed request {n} failed"} + for n in range(20000) + ] + code, output, _ = self.run_script(*container_lines, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertNotIn("pixelfed request", output) + self.assertEqual( + output.splitlines()[-1], + 'docker.service: level=error msg="failed to start container"', + ) + self.assertEqual(len(output.splitlines()), 3) + + def test_a_journal_without_errors_passes(self): + code, _, _ = self.run_script(SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_failing_journalctl_fails_the_check(self): + code, output, _ = self.run_script(rc=1) + self.assertEqual(code, 2) + self.assertIn("journalctl failed with exit code 1", output) + self.assertIn("unrecognized option '--output-fields'", output) + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/cache/README.md b/utils/cache/README.md index 7e42126045..d602b889cd 100644 --- a/utils/cache/README.md +++ b/utils/cache/README.md @@ -26,7 +26,7 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read. -Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. +Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. ## Test-Only Helpers 🧪