diff --git a/docs/contributing/artefact/files/plugins/lookup/applications.md b/docs/contributing/artefact/files/plugins/lookup/applications.md index 67061c3713..46577fc3d7 100644 --- a/docs/contributing/artefact/files/plugins/lookup/applications.md +++ b/docs/contributing/artefact/files/plugins/lookup/applications.md @@ -36,7 +36,8 @@ Ansible forks a fresh worker process for every task. The in-process cache inside The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY`, `SYSTEM_EMAIL_DOMAIN` or `group_names`, or a changed path or content of the tokens file). - The stage-wide task sits in `tasks/stages/01_constructor.yml` after the `networks` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play. -- The carrier is parked again after every role in `tasks/utils/setup/role.yml` and after every persisted token in `roles/sys-token-store/tasks/write.yml`. A re-park whose key still matches serves the carried payload without rendering. +- The carrier is parked again after every role in `tasks/utils/setup/role.yml`, after every nested application load in `tasks/utils/load/app.yml`, and after every persisted token in `roles/sys-token-store/tasks/write.yml`. The first two run on every host of the include, also on hosts the loaded role skips. A re-park whose key still matches serves the carried payload without rendering. +- The action renders with `application_id` removed from its variables, so the carried tree holds the play-scope result of every consumer-aligned `lookup('tls', ...)`, whichever role parks it. - The tokens file enters the key as the SHA-256 of its content, memoised per inode, mtime and size, so a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes. - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`. - User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which read the users cache fresh. diff --git a/docs/requirements/037-corporate-design-overhaul.md b/docs/requirements/037-corporate-design-overhaul.md index 0493056b0a..003583f7c7 100644 --- a/docs/requirements/037-corporate-design-overhaul.md +++ b/docs/requirements/037-corporate-design-overhaul.md @@ -42,7 +42,7 @@ These choices are settled at requirement creation time and bound the implementat - [x] A unit test proves the logo and title precedence: global value, per-role override, and `false`/`0` disabling. - [x] No `roles/*/meta/services.yml` declares a site title as `title`, `titel`, `sitename`, `site_name` or `site_titel` on its primary entity; every site title lives in the role's `design` entry and is read through `lookup('design', application_id)`. - [x] `web-app-joomla` and `web-app-pretix` carry no hardcoded site title. -- [ ] `web-app-decidim` and `web-app-socialhome` honour an inventory title override. Decidim renames an existing organization to an inventory title and back. Open: Socialhome takes the title on its first boot only (`roles/web-app-socialhome/TODO.md`). +- [x] `web-app-decidim` and `web-app-socialhome` honour an inventory title override. Decidim renames an existing organization and Socialhome an existing household to an inventory title and back. - [x] The base gallery on `web-app-dashboard`, `web-app-gitea` and `web-app-nextcloud` is approved by the operator. - [ ] `feature/designer` reaches `main` through the draft PR of `feature/store-i18n` linked below, which merges it round by round. Open: the final designer state is not yet pushed into that PR. @@ -56,7 +56,7 @@ Two base commits changed many roles after their last run: `6c2de017f8` split ven - [x] web-app-baserow - [ ] web-app-bigbluebutton: title not set through Greenlight (the spec asserts the literal upstream name); the meeting client keeps the upstream design; `vars/main.yml` rewritten after the last full run. - [x] web-app-bluesky -- [ ] web-app-bookwyrm: no green complete run on the final state; the last one failed the SeaweedFS avatar upload test. +- [x] web-app-bookwyrm - [x] web-app-checkmk - [x] web-app-chess - [x] web-app-dashboard @@ -65,7 +65,7 @@ Two base commits changed many roles after their last run: `6c2de017f8` split ven - [x] web-app-erpnext - [x] web-app-espocrm - [x] web-app-fediwall -- [ ] web-app-fider: the logo does not reach Fider while the object store is enabled. +- [x] web-app-fider - [ ] web-app-flowise: `env.j2`, `playwright.env.j2` and `tasks/utils/mcp.yml` changed after the last green run. - [x] web-app-friendica - [x] web-app-funkwhale diff --git a/plugins/action/applications_carrier.py b/plugins/action/applications_carrier.py index 93c479ff3d..b57393630b 100644 --- a/plugins/action/applications_carrier.py +++ b/plugins/action/applications_carrier.py @@ -21,7 +21,8 @@ longer match is rejected and the reader renders afresh:: applications_carrier: clear: true -``tasks/utils/setup/role.yml`` parks one after every role and +``tasks/utils/setup/role.yml`` parks one after every role, +``tasks/utils/load/app.yml`` one after every nested application load and ``roles/sys-token-store/tasks/write.yml`` one after every persisted token; where the key still matches they serve the carried payload without rendering. @@ -69,10 +70,17 @@ class ActionModule(ActionBase): return result revalidate_tokens_file(task_vars) - lookup = lookup_loader.get( - "applications", loader=self._loader, templar=self._templar - ) - carrier = lookup.run([], variables=task_vars, carrier=True)[0] + render_vars = dict(task_vars) + render_vars.pop("application_id", None) + previous = self._templar.available_variables + self._templar.available_variables = render_vars + try: + lookup = lookup_loader.get( + "applications", loader=self._loader, templar=self._templar + ) + carrier = lookup.run([], variables=render_vars, carrier=True)[0] + finally: + self._templar.available_variables = previous result.update( changed=False, diff --git a/roles/sys-ctl-hlth-journalctl/README.md b/roles/sys-ctl-hlth-journalctl/README.md index e599a0a870..d43f6e805d 100644 --- a/roles/sys-ctl-hlth-journalctl/README.md +++ b/roles/sys-ctl-hlth-journalctl/README.md @@ -2,16 +2,21 @@ ## Description -Scans `journalctl` over the last day for β€œerror” entries and alerts if any are found. +Scans `journalctl` over the last day for β€œerror” entries outside container logs and alerts if any are found. ## Overview This role searches the systemd journal for errors over the past day and alerts if any are found. +Container output is skipped, because every compose service logs to journald and application errors would otherwise trigger the host check. +Docker daemon and system errors are still reported. ## Features -- Runs `journalctl --since '1 day ago' | grep -i error`. -- Exits non-zero on matches. +- Reads the last day of the journal as JSON with every field in full (`--all`) and drops every entry that carries a `CONTAINER_ID` field. +- Joins a repeated `MESSAGE` field into one line before matching. +- Matches the remaining messages case-insensitively against `error`. +- Prints one `source: message` line per match, naming the systemd unit or, without one, the syslog identifier, so alerts show who logged the error. +- Exits non-zero on matches, and also when `journalctl` itself fails, so a broken check never passes silently. - Scheduled via systemd timer. - Alerts via `sys-ctl-alm-compose` on detection. diff --git a/roles/sys-ctl-hlth-journalctl/files/python/script.py b/roles/sys-ctl-hlth-journalctl/files/python/script.py new file mode 100755 index 0000000000..75ddf4483b --- /dev/null +++ b/roles/sys-ctl-hlth-journalctl/files/python/script.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +import json +import subprocess +import sys +import tempfile + +JOURNALCTL = [ + "journalctl", + "--since", + "1 day ago", + "--no-pager", + "--all", + "--output=json", + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", +] + + +def message_text(value): + """Return a journal MESSAGE as one line of text. + + Args: + value: the MESSAGE field; journald encodes non-UTF-8 messages as a list of bytes and a repeated field as a list of its values. + + Returns: + The message with line breaks collapsed to spaces. + """ + if isinstance(value, list) and all(isinstance(item, int) for item in value): + value = bytes(value).decode("utf-8", errors="replace") + elif isinstance(value, list): + value = "\n".join(message_text(item) for item in value) + return " ".join(str(value).splitlines()) + + +def host_errors(lines): + """Yield `source: message` for every non-container entry that mentions an error. + + Args: + lines: iterable of journalctl JSON lines, one entry per line, consumed lazily. + + Returns: + Iterator of readable error lines. + """ + for line in lines: + if not line.strip(): + continue + entry = json.loads(line) + if "CONTAINER_ID" in entry: + continue + message = message_text(entry.get("MESSAGE", "")) + if "error" not in message.lower(): + continue + source = ( + entry.get("_SYSTEMD_UNIT") or entry.get("SYSLOG_IDENTIFIER") or "unknown" + ) + yield f"{source}: {message}" + + +def main(): + print("Checking journalctl for error messages outside container logs...") + with tempfile.TemporaryFile( + mode="w+", encoding="utf-8", errors="replace" + ) as stderr: + with subprocess.Popen( + JOURNALCTL, + stdout=subprocess.PIPE, + stderr=stderr, + encoding="utf-8", + errors="replace", + ) as journal: + errors = list(host_errors(journal.stdout)) + if journal.returncode != 0: + stderr.seek(0) + print( + f"journalctl failed with exit code {journal.returncode}: {stderr.read().strip()}" + ) + return 2 + if errors: + print("Some errors were found:") + print("\n".join(errors)) + return 1 + print("No errors found in the journal.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh b/roles/sys-ctl-hlth-journalctl/files/shell/script.sh deleted file mode 100644 index 4903b597d1..0000000000 --- a/roles/sys-ctl-hlth-journalctl/files/shell/script.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -echo "Checking journalctl for error messages..." -journalctl_errors="$(journalctl --since '1 day ago' --no-pager | grep -i 'error')" -if [ -n "$journalctl_errors" ] -then - echo "Some errors where found: $journalctl_errors" - exit 1 -fi -echo "All docker containers are healthy." -exit 0 diff --git a/roles/web-app-fider/TODO.md b/roles/web-app-fider/TODO.md index 17f69111c3..c5eb7813a8 100644 --- a/roles/web-app-fider/TODO.md +++ b/roles/web-app-fider/TODO.md @@ -1,4 +1,3 @@ # TODO - With `services.sso.enabled` false the role creates no tenant: `tasks/00_core.yml` includes `tasks/01_setup_oidc.yml` (tenant bootstrap, administrator, activation, provider row) only under `FIDER_OIDC_ENABLED`. Upstream then redirects every route to its installation form `/signup`, which any visitor can submit (read from the task conditions and from `app/middlewares/tenant.go` and `app/handlers/signup.go` of the deployed commit `67c52826`; the compose stack of the design pass ran with `sso` on). Fider has no password sign-in, so such a deployment also has no account. Options: bootstrap tenant and administrator in every variant and keep only the provider row under the flag; or refuse `/signup` at the proxy while no tenant exists. -- With the object store enabled (`BLOB_STORAGE=s3`) the board keeps upstream's favicon and shows no logo, and the sign-in dialog loads `https://login.fider.io/static/assets/logo.png` from the browser (measured on the compose stack before a logo was set: the image rendered in the dialog). Cause: `tasks/02_design.yml` places the logo in the `blobs` table, which the S3 blob store never reads, so `FIDER_DESIGN_LOGO_ENABLED` is false there and `tenants.logo_bkey` stays empty; upstream's upload route `POST /_api/admin/settings/general` needs a session cookie. Options: put the object `tenants//logos/infinito-design-logo.png` into the role's bucket with the role's object store credentials and set `logo_bkey`; or upload through the app's API with a session obtained from the identity provider. diff --git a/roles/web-app-fider/files/playwright/test-design.js b/roles/web-app-fider/files/playwright/test-design.js index 9ce340c11b..6475ad4612 100644 --- a/roles/web-app-fider/files/playwright/test-design.js +++ b/roles/web-app-fider/files/playwright/test-design.js @@ -9,7 +9,7 @@ const { galleryEnabled, tokenValue, } = require("./design"); -const { decodeDotenvQuotedValue, gotoOnion, performKeycloakLoginForm, requireDotenvValue } = require("./personas"); +const { apiGetOnion, decodeDotenvQuotedValue, gotoOnion, performKeycloakLoginForm, requireDotenvValue } = require("./personas"); const { isServiceEnabled, skipUnlessServiceEnabled } = require("./service-gating"); const { resolveTimeout } = require("./timeouts"); const { clickFiderSsoButton } = require("./_shared"); @@ -17,8 +17,6 @@ const { clickFiderSsoButton } = require("./_shared"); const base = requireDotenvValue(process.env.FIDER_BASE_URL, "FIDER_BASE_URL").replace(/\/$/, ""); const adminUsername = requireDotenvValue(process.env.ADMIN_USERNAME, "ADMIN_USERNAME"); const adminPassword = requireDotenvValue(process.env.ADMIN_PASSWORD, "ADMIN_PASSWORD"); -const title = decodeDotenvQuotedValue(process.env.DESIGN_TITLE || ""); -const logoEnabled = process.env.DESIGN_LOGO_ENABLED === "true"; const MODES = ["light", "dark"]; const READY_MS = 60_000; @@ -270,14 +268,18 @@ exports.register = function () { }); test("design: the board carries the configured title", async ({ page }) => { + const title = decodeDotenvQuotedValue(process.env.DESIGN_TITLE || ""); + test.skip(!title, "title replacement is disabled for this role"); await openHome(page); - expect(title, "DESIGN_TITLE must be set in the Playwright env file").toBeTruthy(); await expect(page).toHaveTitle(title); await expect(page.locator(`${BRAND} h1`)).toHaveText(title); }); - test("design: logo and favicon are the configured ones", async ({ page }) => { - test.skip(!logoEnabled, "the role places the logo in the database blob store only; with the object store enabled the tenant keeps its own logo"); + test("design: logo and favicon are the configured ones", async ({ page, request }) => { + test.skip(process.env.DESIGN_LOGO_ENABLED !== "true", "logo replacement is disabled for this role"); + const image = await apiGetOnion(request, `${base}/static/images/logos/infinito-design-logo.png?size=100`); + expect(image.status(), "the logo blob is served from the active blob store").toBe(200); + expect(image.headers()["content-type"]).toBe("image/png"); await openHome(page); const logo = page.locator(`${BRAND} img`); await expect(logo).toBeVisible(); @@ -288,6 +290,10 @@ exports.register = function () { const brandBox = await page.locator(BRAND).boundingBox(); expect(brandBox.width, "logo and title form a wide brand").toBeGreaterThan(brandBox.height); await expect(page.locator("link[rel='icon']").first()).toHaveAttribute("href", LOGO_FAVICON); + await page.locator(PRIMARY).first().click(); + await modalOpened(page); + await expect(page.locator(`${MODAL} img`).first()).toHaveAttribute("src", LOGO_IMAGE); + await expect(page.locator(`${MODAL} img[src*='login.fider.io']`)).toHaveCount(0); }); test("design: the board follows the browser until the switch is used, then the switch wins", async ({ page }) => { diff --git a/roles/web-app-fider/files/playwright/test-seaweedfs.js b/roles/web-app-fider/files/playwright/test-seaweedfs.js index f87b8a456f..58178ba1b3 100644 --- a/roles/web-app-fider/files/playwright/test-seaweedfs.js +++ b/roles/web-app-fider/files/playwright/test-seaweedfs.js @@ -1,16 +1,3 @@ -// SeaweedFS object-store scenario for Fider. -// -// Fider is configured with the S3 blob backend (env.j2: BLOB_STORAGE=s3 + -// BLOB_STORAGE_S3_BUCKET / _ACCESS_KEY_ID / _SECRET_ACCESS_KEY / -// _ENDPOINT_URL / _REGION), so a tenant logo the administrator uploads on the -// admin General settings page is written to the consumer bucket as a new -// private blob (Fider stores blobs with ACL=private and reverse-proxies them -// through its own /images route, which is why the role declares no public -// flag). The action signs the administrator in through Fider's SSO button + -// Keycloak form, opens the admin General settings page, sets the "Your Logo" -// file input to a small PNG and clicks Save; the shared check proves the -// bucket grew via the Filer UI. -// // Required env (rendered by templates/playwright.env.j2): // FIDER_BASE_URL, the OIDC login vars consumed by performKeycloakLoginForm, // and the SEAWEEDFS_* keys consumed by runSeaweedfsStorageCheck. @@ -19,34 +6,21 @@ const { test, expect } = require("@playwright/test"); const { resolveTimeout, isOnionTarget } = require("./timeouts"); const { skipUnlessServiceEnabled } = require("./service-gating"); const { runSeaweedfsStorageCheck, performKeycloakLoginForm, decodeDotenvQuotedValue } = require("./personas"); +const { clickFiderSsoButton } = require("./_shared"); -const LOGO_PNG = Buffer.from( +const ATTACHMENT_PNG_BASE64 = "iVBORw0KGgoAAAANSUhEUgAAAMgAAADICAIAAAAiOjnJAAABeElEQVR42u3SMQ0AAAgEsVeHMDQhEBMM" + - "DE2q4HKpHjgXCTAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + - "scBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAsjIWxwFgYC2OBsTAWxgJjYSyMBcbC" + - "WBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyFsVTAWBgLY4GxMBbGAmNhLIwFxsJY" + - "GAuMhbEwFhgLY2EsMBbGwlhgLIyFscBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAs" + - "jIWxwFgYC2OBsTAWxgJjYSyMBcbCWBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyF" + - "sVTAWBgLY4GxMBbGAmNhLIwFxsJYGAtjqYCxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + - "scBYGAtjgbH4ZgHTqvyKw+KRzwAAAABJRU5ErkJggg==", - "base64", -); - -async function clickFiderSsoButton(locator) { - const signInLink = locator - .getByRole("button", { name: /sign in/i }) - .or(locator.getByRole("link", { name: /sign in/i })); - await signInLink.first().waitFor({ state: "visible", timeout: resolveTimeout(30_000) }); - await signInLink.first().click(); - - const ssoButton = locator.getByRole("link", { name: /continue with/i }); - await ssoButton.first().waitFor({ state: "visible", timeout: resolveTimeout(15_000) }); - await ssoButton.first().click({ force: true, timeout: resolveTimeout(30_000) }); -} + "DE2q4HKpHjgXCTAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + + "scBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAsjIWxwFgYC2OBsTAWxgJjYSyMBcbC" + + "WBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyFsVTAWBgLY4GxMBbGAmNhLIwFxsJY" + + "GAuMhbEwFhgLY2EsMBbGwlhgLIyFscBYGAtjgbEwFsYCY2EsjAXGwlgYC4yFsTAWGAtjYSwwFsbCWGAs" + + "jIWxwFgYC2OBsTAWxgJjYSyMBcbCWBgLjIWxMBYYC2NhLDAWxsJYYCyMhbHAWBgLY4GxMBbGAmNhLIyF" + + "sVTAWBgLY4GxMBbGAmNhLIwFxsJYGAtjqYCxMBbGAmNhLIwFxsJYGAuMhbEwFhgLY2EsMBbGwlhgLIyF" + + "scBYGAtjgbH4ZgHTqvyKw+KRzwAAAABJRU5ErkJggg=="; test.use({ ignoreHTTPSErrors: true }); -test("seaweedfs: an uploaded Fider tenant logo is stored in the SeaweedFS bucket", async ({ page, browser }) => { +test("seaweedfs: a Fider post attachment is stored in the SeaweedFS bucket", async ({ page, browser }) => { test.skip(isOnionTarget(), "SeaweedFS filer UI is not a Tor surface on an onion node (headless backend)"); skipUnlessServiceEnabled("seaweedfs"); test.setTimeout(resolveTimeout(180_000)); @@ -54,67 +28,52 @@ test("seaweedfs: an uploaded Fider tenant logo is stored in the SeaweedFS bucket const fiderBaseUrl = decodeDotenvQuotedValue(process.env.FIDER_BASE_URL || ""); const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); + const marker = `Storage check ${Date.now()}`; + let post = null; await runSeaweedfsStorageCheck(page, browser, { - label: "a Fider admin tenant-logo upload", - expectInPlaceOverwrite: true, + label: "a Fider post attachment", action: async (appPage) => { expect(fiderBaseUrl, "FIDER_BASE_URL must be set for the Fider seaweedfs check").toBeTruthy(); expect(adminUsername, "ADMIN_USERNAME must be set for the Fider seaweedfs check").toBeTruthy(); expect(adminPassword, "ADMIN_PASSWORD must be set for the Fider seaweedfs check").toBeTruthy(); - const baseUrl = fiderBaseUrl.replace(/\/$/, ""); - - await appPage.goto(`${baseUrl}/`, { waitUntil: "domcontentloaded" }); + await appPage.goto(`${fiderBaseUrl.replace(/\/$/, "")}/`, { waitUntil: "domcontentloaded" }); await clickFiderSsoButton(appPage); await performKeycloakLoginForm(appPage, adminUsername, adminPassword); - await expect( appPage.locator(".c-menu-user").first(), - "the administrator must be authenticated before opening the admin settings", + "the administrator must be authenticated before creating the post", ).toBeVisible({ timeout: resolveTimeout(60_000) }); - await appPage.goto(`${baseUrl}/admin`, { waitUntil: "domcontentloaded" }); - - const fileInput = appPage.locator('input[type="file"][accept="image/*"]').first(); - await expect( - fileInput, - "the Fider admin General settings page must expose the 'Your Logo' image input", - ).toBeAttached({ timeout: resolveTimeout(60_000) }); - - const marker = `infinito-storage-check-${Date.now()}.png`; - await fileInput.setInputFiles({ - name: marker, - mimeType: "image/png", - buffer: LOGO_PNG, - }); - - await expect( - appPage.locator('.c-image-upload img[src^="data:image"], .preview img[src^="data:image"]').first(), - "the selected logo must be loaded into the form (preview rendered) before saving", - ).toBeVisible({ timeout: resolveTimeout(30_000) }); - - const saveButton = appPage.getByRole("button", { name: /^save$/i }).first(); - await expect( - saveButton, - "the Fider admin General settings page must expose a Save action that persists the logo", - ).toBeVisible({ timeout: resolveTimeout(60_000) }); - const [settingsResp] = await Promise.all([ - appPage.waitForResponse( - (r) => /\/_api\/admin\/settings\/general/.test(r.url()) && r.request().method() === "POST", - { timeout: resolveTimeout(60_000) }, - ), - saveButton.click(), - ]); - expect( - settingsResp.ok(), - `Fider must accept the General-settings save that persists the logo (got HTTP ${settingsResp.status()})`, - ).toBeTruthy(); - const reqBody = settingsResp.request().postData() || ""; - expect( - reqBody, - "the General-settings save must carry the logo upload payload (empty upload = silent no-op)", - ).toMatch(/"upload"\s*:\s*\{[^}]*"content"/); + post = await appPage.evaluate( + async ({ title, content }) => { + const response = await fetch("/api/v1/posts", { + method: "POST", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + credentials: "same-origin", + body: JSON.stringify({ + title, + description: "A post that proves attachments reach the object store.", + attachments: [{ upload: { fileName: "storage-check.png", contentType: "image/png", content } }], + }), + }); + return { status: response.status, body: await response.json().catch(() => ({})) }; + }, + { title: marker, content: ATTACHMENT_PNG_BASE64 }, + ); + expect(post.status, "Fider must accept the post with its attachment").toBe(200); }, }); + + const removal = await page.evaluate(async (number) => { + const response = await fetch(`/api/v1/posts/${number}`, { + method: "DELETE", + headers: { Accept: "application/json", "Content-Type": "application/json" }, + credentials: "same-origin", + body: JSON.stringify({ text: "" }), + }); + return response.status; + }, post.body.number); + expect(removal, "the storage-check post is removed again").toBe(200); }); diff --git a/roles/web-app-fider/files/shell/design-logo-object.sh b/roles/web-app-fider/files/shell/design-logo-object.sh new file mode 100644 index 0000000000..5d919eebe4 --- /dev/null +++ b/roles/web-app-fider/files/shell/design-logo-object.sh @@ -0,0 +1,32 @@ +#!/bin/sh +# Args: action (put|delete), object URL, S3 region. +# Stdin: ":" on the first line; for put, the PNG as base64 after it. +# Prints CHANGED, UNCHANGED, REMOVED or ABSENT. +set -eu +action=$1 +url=$2 +region=$3 +IFS= read -r credentials +config() { + printf 'user = "%s"\naws-sigv4 = "aws:amz:%s:s3"\nurl = "%s"\nsilent\nfail\n' "$credentials" "$region" "$url" +} +if [ "$action" = delete ]; then + if config | curl -K - --head -o /dev/null; then + config | curl -K - -X DELETE -o /dev/null + echo REMOVED + else + echo ABSENT + fi + exit 0 +fi +body=$(mktemp) +trap 'rm -f "$body"' EXIT +base64 -d > "$body" +wanted=$(md5sum < "$body" | cut -d' ' -f1) +current=$(config | curl -K - | md5sum | cut -d' ' -f1) +if [ "$current" = "$wanted" ]; then + echo UNCHANGED + exit 0 +fi +config | curl -K - -X PUT -H 'Content-Type: image/png' --data-binary "@$body" -o /dev/null +echo CHANGED diff --git a/roles/web-app-fider/files/sql/apply-design.sql b/roles/web-app-fider/files/sql/apply-design.sql index 590f8a9059..bd28fb9754 100644 --- a/roles/web-app-fider/files/sql/apply-design.sql +++ b/roles/web-app-fider/files/sql/apply-design.sql @@ -1,4 +1,4 @@ --- Args: title (empty keeps the name), bootstrap_name (the name the role bootstrapped the tenant with), css, logo_key, logo_base64 (empty keeps the logo). +-- Args: title (empty keeps the name), bootstrap_name (the name the role bootstrapped the tenant with), css, logo_key, logo_base64 (non-empty writes the logo into the blobs table), logo_stored (non-empty when the logo is in the active blob store). -- Returns one row: the number of rows that changed. WITH tenant AS ( SELECT id FROM tenants ORDER BY id LIMIT 1 @@ -18,7 +18,7 @@ WITH tenant AS ( THEN %(title)s ELSE tenants.name END AS name, CASE WHEN tenants.custom_css = '' OR position('--infinito-design-carrier' IN tenants.custom_css) > 0 THEN %(css)s ELSE tenants.custom_css END AS custom_css, - CASE WHEN %(logo_base64)s <> '' AND COALESCE(tenants.logo_bkey, '') = '' + CASE WHEN %(logo_stored)s <> '' AND COALESCE(tenants.logo_bkey, '') = '' THEN %(logo_key)s ELSE tenants.logo_bkey END AS logo_bkey FROM tenants JOIN tenant ON tenant.id = tenants.id diff --git a/roles/web-app-fider/files/sql/select-tenant-id.sql b/roles/web-app-fider/files/sql/select-tenant-id.sql new file mode 100644 index 0000000000..35eb9c73b9 --- /dev/null +++ b/roles/web-app-fider/files/sql/select-tenant-id.sql @@ -0,0 +1,2 @@ +-- Returns one row: the id of the tenant the role designs, 0 while no tenant exists. +SELECT COALESCE(MIN(id), 0) FROM tenants; diff --git a/roles/web-app-fider/tasks/02_design.yml b/roles/web-app-fider/tasks/02_design.yml index aa09401cac..7ac9c5f7bd 100644 --- a/roles/web-app-fider/tasks/02_design.yml +++ b/roles/web-app-fider/tasks/02_design.yml @@ -17,6 +17,34 @@ register: fider_design_logo when: FIDER_DESIGN_LOGO_ENABLED | bool +- name: "πŸͺ£ Fider | Logo in the object store" + when: FIDER_OBJSTORE_ENABLED | bool + block: + - name: "πŸͺ£ Fider | Read the tenant id" + database_query: + config: "{{ lookup('database', application_id) }}" + query_file: "{{ role_path }}/files/sql/select-tenant-id.sql" + expect_rows: true + mask_values: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + register: fider_design_tenant + changed_when: false + + - name: "πŸͺ£ Fider | Store or remove the generated logo in the bucket" + ansible.builtin.shell: > + container exec -i {{ FIDER_OBJSTORE_CONTAINER_ADDRESS }} + sh -c {{ lookup('file', 'shell/design-logo-object.sh') | quote }} design-logo-object + {{ 'put' if FIDER_DESIGN_LOGO_ENABLED | bool else 'delete' }} + {{ (lookup('objstore', application_id, 'url') ~ '/' ~ lookup('objstore', application_id, 'bucket') ~ '/tenants/' ~ fider_design_tenant.rows[0][0] ~ '/' ~ + FIDER_DESIGN_LOGO_KEY) | quote }} + {{ lookup('objstore', application_id, 'region') | quote }} + args: + stdin: "{{ lookup('objstore', application_id, 'access_key') }}:{{ lookup('objstore', application_id, 'secret_key') }}\n{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED + | bool else '' }}" + register: fider_design_logo_object + changed_when: fider_design_logo_object.stdout in ['CHANGED', 'REMOVED'] + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: fider_design_tenant.rows[0][0] | int > 0 + - name: "🎨 Fider | Apply title, custom CSS and logo" database_query: config: "{{ lookup('database', application_id) }}" @@ -26,7 +54,8 @@ bootstrap_name: "{{ SOFTWARE_NAME }}" css: "{{ lookup('template', 'custom.css.j2') }}" logo_key: "{{ FIDER_DESIGN_LOGO_KEY }}" - logo_base64: "{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED | bool else '' }}" + logo_base64: "{{ fider_design_logo.content if FIDER_DESIGN_LOGO_ENABLED | bool and not (FIDER_OBJSTORE_ENABLED | bool) else '' }}" + logo_stored: "{{ 'true' if FIDER_DESIGN_LOGO_ENABLED | bool else '' }}" expect_rows: true mask_values: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" register: fider_design_apply diff --git a/roles/web-app-fider/vars/main.yml b/roles/web-app-fider/vars/main.yml index 111d4a286d..76e9bc7f23 100644 --- a/roles/web-app-fider/vars/main.yml +++ b/roles/web-app-fider/vars/main.yml @@ -8,9 +8,10 @@ FIDER_OIDC_ENABLED: "{{ lookup('config', application_id, 'services.sso.enabled') FIDER_DESIGN_ENABLED: "{{ lookup('config', application_id, 'services.design.enabled') | bool }}" FIDER_DESIGN_TITLE: "{{ lookup('design', application_id).title if FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).title is string else '' }}" -FIDER_DESIGN_LOGO_ENABLED: "{{ FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).logo is string and not (lookup('objstore', application_id, 'enabled') - | bool) }}" +FIDER_DESIGN_LOGO_ENABLED: "{{ FIDER_DESIGN_ENABLED | bool and lookup('design', application_id).logo is string }}" FIDER_DESIGN_LOGO_KEY: "logos/infinito-design-logo.png" +FIDER_OBJSTORE_ENABLED: "{{ lookup('objstore', application_id, 'enabled') | bool }}" +FIDER_OBJSTORE_CONTAINER_ADDRESS: "{{ lookup('container_address', lookup('objstore', application_id, 'id'), lookup('objstore', application_id, 'engine')) }}" FIDER_TENANT_BOOTSTRAP_BODY: tenantName: "{{ SOFTWARE_NAME }}" diff --git a/roles/web-app-socialhome/TODO.md b/roles/web-app-socialhome/TODO.md index c513b57909..e0f76285a9 100644 --- a/roles/web-app-socialhome/TODO.md +++ b/roles/web-app-socialhome/TODO.md @@ -40,7 +40,6 @@ Open items that only a live run can settle: ## Open from the design pass -- Title override: `SH_INSTANCE_NAME` reaches the home name only on the first boot, because `ensure_instance_identity` (upstream `socialhome/identity_bootstrap.py`) writes `display_name` once and `GET /api/instance/config` prefers that row; an inventory title set later changes the env but not the shown name. No route writes that row at 2026.10.8 (measured: `routes/instance.py` only reads `federation_repo.get_local_identity()`, and no `set`/`update` of the local identity exists in the package), and the row is the identity peers receive. Options: ask upstream for an admin route that renames the household, or accept first-boot only. - Shared sign-in budget: `POST /api/auth/token` allows 5 attempts per 15 minutes per `request.remote`, and the app ignores `X-Forwarded-For`, so every browser behind the front proxy and the deploy's own token request share one bucket (measured: all logins in `/api/admin/auth-audit` come from `192.168.75.1`). At 2026.10.11 the budget is the module constant `AUTH_TOKEN_RATE_LIMIT` in `socialhome/routes/users.py` with no env or TOML key, and `trusted_proxies` exists for the GFS only (`socialhome/global_server/config.py`); the design spec keeps one administrator token in the Playwright staging volume. Options: let the proxy reach the app with the client address (PROXY protocol or upstream trusted-proxy support), or ask upstream to honour a trusted forwarded header. - `tasks/01_setup.yml` comment is stale: at 2026.10.8 the headless seed marks setup complete itself (upstream `platform/standalone/adapter.py`), so the task always answers 409 (measured: `ok`, never `changed`). Options: drop the task or rewrite the comment. - `meta/csp.yml` keeps `script-src-elem unsafe-inline` for an inline theme script the 2026.10.8 shell no longer has (it loads `assets/theme-boot.js`; measured: the served `/feed` carries no inline script of its own). Options: drop the flag after checking `hlth-csp`; the injected design script then needs its CSP hash, which `sys-front-inj-design` adds. diff --git a/roles/web-app-socialhome/files/playwright/test-design.js b/roles/web-app-socialhome/files/playwright/test-design.js index 9032444377..224c9c7a82 100644 --- a/roles/web-app-socialhome/files/playwright/test-design.js +++ b/roles/web-app-socialhome/files/playwright/test-design.js @@ -570,6 +570,12 @@ exports.register = function () { } await expect(page.locator("link[rel~='icon']").first()).toHaveAttribute("href", faviconUrl); } + if (title) { + const config = await apiGetOnion(page.request, `${base}/api/instance/config`); + expect((await config.json()).instance_name, "the household carries the configured title").toBe(title); + await visit(page, `${base}/feed`); + await expect(page.locator(TOPBAR_TITLE), "the feed is named after the household").toHaveText(title); + } }); test("design: the sign-in page shows the generated logo", async ({ page }) => { diff --git a/roles/web-app-socialhome/tasks/02_design.yml b/roles/web-app-socialhome/tasks/02_design.yml index b75ea6f514..c133fae330 100644 --- a/roles/web-app-socialhome/tasks/02_design.yml +++ b/roles/web-app-socialhome/tasks/02_design.yml @@ -5,49 +5,20 @@ checksum_algorithm: sha1 register: socialhome_theme_marker -- name: "🎨 Converge the household primary colour" - when: SOCIALHOME_DESIGN_ENABLED | bool or socialhome_theme_marker.stat.exists - block: - - name: "πŸ”‘ Request an administrator token for the household theme" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/auth/token" - method: POST - body_format: json - body: - username: "{{ SOCIALHOME_ADMIN_USERNAME }}" - password: "{{ SOCIALHOME_ADMIN_PASSWORD }}" - status_code: 200 - retries: 0 - register: socialhome_theme_token - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "πŸ“₯ Look up the household name the role wrote last" + ansible.builtin.stat: + path: "{{ SOCIALHOME_NAME_STATE_FILE }}" + register: socialhome_name_marker - - name: "🎨 Read the household theme" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" - method: GET - headers: - Authorization: "Bearer {{ socialhome_theme_token.json.token }}" - status_code: 200 - register: socialhome_theme - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "πŸ“₯ Read the household name the role wrote last" + ansible.builtin.slurp: + src: "{{ SOCIALHOME_NAME_STATE_FILE }}" + register: socialhome_name_state_file + when: socialhome_name_marker.stat.exists - - name: "🎨 Write the household primary colour" - uri_retry: - url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" - method: PUT - body_format: json - body: - primary_color: "{{ SOCIALHOME_THEME_PRIMARY }}" - headers: - Authorization: "Bearer {{ socialhome_theme_token.json.token }}" - status_code: 200 - register: socialhome_theme_written - changed_when: true - when: - - socialhome_theme.json.primary_color | lower != SOCIALHOME_THEME_PRIMARY - - socialhome_theme.json.primary_color | lower == SOCIALHOME_BRAND_PRIMARY or (socialhome_theme_marker.stat.exists and - socialhome_theme_marker.stat.checksum == (socialhome_theme.json.primary_color | lower | hash('sha1'))) - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "🎨 Converge the household primary colour and name" + include_tasks: 02_design/converge.yml + when: SOCIALHOME_DESIGN_ENABLED | bool or socialhome_theme_marker.stat.exists or socialhome_name_marker.stat.exists - name: "🏷️ Mark the household primary colour as the role's" stack_host_copy: @@ -58,8 +29,24 @@ - SOCIALHOME_DESIGN_ENABLED | bool - socialhome_theme_written is changed +- name: "🏷️ Mark the household name as the role's" + stack_host_copy: + content: "{{ {'applied': SOCIALHOME_DESIGN_TITLE, 'previous': (socialhome_name_state.previous if socialhome_name_state | length > 0 else socialhome_name_current)} + | to_json }}" + dest: "{{ SOCIALHOME_NAME_STATE_FILE }}" + mode: "0644" + when: + - SOCIALHOME_DESIGN_TITLE | length > 0 + - socialhome_name_written is changed or (socialhome_name_state | length == 0 and socialhome_name_current == SOCIALHOME_DESIGN_TITLE) + - name: "🧹 Drop the household theme marker" ansible.builtin.file: path: "{{ SOCIALHOME_THEME_MARKER_FILE }}" state: absent when: not (SOCIALHOME_DESIGN_ENABLED | bool) + +- name: "🧹 Drop the household name marker" + ansible.builtin.file: + path: "{{ SOCIALHOME_NAME_STATE_FILE }}" + state: absent + when: SOCIALHOME_DESIGN_TITLE | length == 0 diff --git a/roles/web-app-socialhome/tasks/02_design/converge.yml b/roles/web-app-socialhome/tasks/02_design/converge.yml new file mode 100644 index 0000000000..ac72fc6129 --- /dev/null +++ b/roles/web-app-socialhome/tasks/02_design/converge.yml @@ -0,0 +1,73 @@ +--- +- name: "πŸ”‘ Request an administrator token for the household theme" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/auth/token" + method: POST + body_format: json + body: + username: "{{ SOCIALHOME_ADMIN_USERNAME }}" + password: "{{ SOCIALHOME_ADMIN_PASSWORD }}" + status_code: 200 + register: socialhome_theme_token + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🎨 Read the household theme" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" + method: GET + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + register: socialhome_theme + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🎨 Write the household primary colour" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/theme" + method: PUT + body_format: json + body: + primary_color: "{{ SOCIALHOME_THEME_PRIMARY }}" + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + register: socialhome_theme_written + changed_when: true + when: + - socialhome_theme.json.primary_color | lower != SOCIALHOME_THEME_PRIMARY + - socialhome_theme.json.primary_color | lower == SOCIALHOME_BRAND_PRIMARY or (socialhome_theme_marker.stat.exists and socialhome_theme_marker.stat.checksum + == (socialhome_theme.json.primary_color | lower | hash('sha1'))) + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🏷️ Read the household name" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/instance/config" + method: GET + status_code: 200 + register: socialhome_instance_config + +- name: "🏷️ Hold the household name and the name the role wrote last" + ansible.builtin.set_fact: + socialhome_name_current: "{{ socialhome_instance_config.json.instance_name }}" + socialhome_name_state: "{{ (socialhome_name_state_file.content | b64decode | from_json) if socialhome_name_marker.stat.exists else {} }}" + +- name: "🏷️ Rename the household" + uri_retry: + url: "{{ SOCIALHOME_INTERNAL_URL }}/api/admin/instance" + method: PATCH + body_format: json + body: + display_name: "{{ socialhome_name_target }}" + headers: + Authorization: "Bearer {{ socialhome_theme_token.json.token }}" + status_code: 200 + vars: + socialhome_name_ours: "{{ socialhome_name_state | length > 0 and socialhome_name_current == socialhome_name_state.applied }}" + socialhome_name_target: >- + {{ (socialhome_name_state.previous if socialhome_name_ours | bool else '') if SOCIALHOME_DESIGN_TITLE | length == 0 + else ('' if (socialhome_name_state | length > 0 and not (socialhome_name_ours | bool)) or socialhome_name_current == SOCIALHOME_DESIGN_TITLE + else SOCIALHOME_DESIGN_TITLE) }} + register: socialhome_name_written + changed_when: true + when: socialhome_name_target | length > 0 + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" diff --git a/roles/web-app-socialhome/tasks/main.yml b/roles/web-app-socialhome/tasks/main.yml index 17fb8f576a..c13e243e8f 100644 --- a/roles/web-app-socialhome/tasks/main.yml +++ b/roles/web-app-socialhome/tasks/main.yml @@ -22,5 +22,5 @@ - name: "πŸ” Complete the standalone first-boot setup for '{{ application_id }}'" include_tasks: 01_setup.yml - - name: "🎨 Converge the household theme for '{{ application_id }}'" + - name: "🎨 Converge the household theme and name for '{{ application_id }}'" include_tasks: 02_design.yml diff --git a/roles/web-app-socialhome/vars/main.yml b/roles/web-app-socialhome/vars/main.yml index f1110a1657..5ef27dad5c 100644 --- a/roles/web-app-socialhome/vars/main.yml +++ b/roles/web-app-socialhome/vars/main.yml @@ -17,6 +17,9 @@ SOCIALHOME_CONTAINER_ADDRESS: "{{ lookup('container_address', application_id, en SOCIALHOME_MAX_STORAGE_BYTES: 4294967296 SOCIALHOME_DESIGN_ENABLED: "{{ lookup('config', application_id, 'services.design.enabled') | bool }}" +SOCIALHOME_DESIGN_TITLE: "{{ lookup('design', application_id).title if SOCIALHOME_DESIGN_ENABLED | bool and lookup('design', application_id).title is string else + '' }}" +SOCIALHOME_NAME_STATE_FILE: "{{ [lookup('container', application_id, 'directories.volumes'), 'household-name.json'] | path_join }}" SOCIALHOME_BRAND_PRIMARY: "#d2542a" SOCIALHOME_THEME_PRIMARY: >- {{ (lookup('design_palette', lookup('config', 'web-svc-design', 'services.design.colors.base')).light['--design-primary'] | lower) diff --git a/tasks/utils/load/app.yml b/tasks/utils/load/app.yml index abf1e21cb4..818b4d6278 100644 --- a/tasks/utils/load/app.yml +++ b/tasks/utils/load/app.yml @@ -15,3 +15,6 @@ domain: "{{ lookup('domain', load_app_id) }}" when: - groups[load_app_id] is not defined or inventory_hostname in groups[load_app_id] + +- name: "🧊 Carry the applications SPOT past nested {{ load_app_id }}" + applications_carrier: {} diff --git a/tests/unit/python/plugins/action/test_applications_carrier.py b/tests/unit/python/plugins/action/test_applications_carrier.py index 9f69c1d1d5..4e0c8d213e 100644 --- a/tests/unit/python/plugins/action/test_applications_carrier.py +++ b/tests/unit/python/plugins/action/test_applications_carrier.py @@ -12,21 +12,30 @@ class _FakeTask: self.args = {} if args is None else dict(args) +class _FakeTemplar: + def __init__(self, available_variables=None): + self.available_variables = available_variables + + class _FakeLookup: - def __init__(self, carrier): + def __init__(self, carrier, templar=None): self.carrier = carrier + self.templar = templar self.calls = [] + self.templar_variables = [] def run(self, terms, variables=None, **kwargs): self.calls.append((terms, variables, kwargs)) + if self.templar is not None: + self.templar_variables.append(dict(self.templar.available_variables)) return [self.carrier] -def _make_action(task: _FakeTask) -> ActionModule: +def _make_action(task: _FakeTask, templar=None) -> ActionModule: action = object.__new__(ActionModule) action._task = task action._loader = None - action._templar = None + action._templar = _FakeTemplar() if templar is None else templar return action @@ -56,6 +65,31 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): self.assertEqual(variables, {"applications": {}}) self.assertTrue(kwargs["carrier"]) + def test_renders_without_the_callers_application_id(self): + task_vars = {"applications": {}, "application_id": "web-app-bluesky"} + templar = _FakeTemplar(task_vars) + lookup = _FakeLookup({"key": ["roles", []], "applications": {}}, templar) + action = _make_action(_FakeTask(), templar) + with ( + patch( + "plugins.action.applications_carrier.ActionBase.run", + autospec=True, + return_value={}, + ), + patch( + "plugins.action.applications_carrier.lookup_loader.get", + return_value=lookup, + ), + patch("plugins.action.applications_carrier.revalidate_tokens_file"), + ): + action.run(task_vars=task_vars) + + _, variables, _ = lookup.calls[0] + self.assertEqual(variables, {"applications": {}}) + self.assertEqual(lookup.templar_variables, [{"applications": {}}]) + self.assertIs(templar.available_variables, task_vars) + self.assertEqual(task_vars["application_id"], "web-app-bluesky") + def test_revalidates_the_tokens_file_before_rendering(self): order = [] lookup = _FakeLookup({"key": ["roles", []], "applications": {}}) diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py new file mode 100644 index 0000000000..8a4f44007f --- /dev/null +++ b/tests/unit/python/roles/sys-ctl-hlth-journalctl/files/test_script.py @@ -0,0 +1,147 @@ +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from utils import PROJECT_ROOT + +SCRIPT = PROJECT_ROOT / "roles/sys-ctl-hlth-journalctl/files/python/script.py" + +JOURNALCTL_STUB = """#!/usr/bin/env bash +printf '%s\\n' "$*" >"${STUB_DIR}/args" +cat "${STUB_DIR}/journal" +if [ "${STUB_RC}" -ne 0 ]; then +\techo "journalctl: unrecognized option '--output-fields'" >&2 +fi +exit "${STUB_RC}" +""" + +APP_ERROR = { + "MESSAGE": "ERROR: pixelfed queue failed", + "CONTAINER_ID": "0123456789ab", + "SYSLOG_IDENTIFIER": "pixelfed", + "_SYSTEMD_UNIT": "docker.service", +} +DAEMON_ERROR = { + "MESSAGE": 'level=error msg="failed to start container"', + "SYSLOG_IDENTIFIER": "dockerd", + "_SYSTEMD_UNIT": "docker.service", +} +KERNEL_ERROR = {"MESSAGE": "EXT4-fs error (device sda1)", "SYSLOG_IDENTIFIER": "kernel"} +BINARY_ERROR = { + "MESSAGE": list(b"I/O error \xff on disk"), + "_SYSTEMD_UNIT": "smartd.service", +} +REPEATED_ERROR = { + "MESSAGE": ["unit started", list(b"disk error \xff on sdb")], + "_SYSTEMD_UNIT": "zfs-import.service", +} +SYSTEM_INFO = { + "MESSAGE": "Started Daily apt download activities.", + "_SYSTEMD_UNIT": "init.scope", +} + + +class TestHlthJournalctlScript(unittest.TestCase): + def run_script(self, *entries, rc=0): + """Run the health script against a stubbed journalctl. + + Args: + entries: journal entries the stub prints, one JSON object per line. + rc: exit code of the stubbed journalctl. + + Returns: + Tuple of the exit code, the script output, and the journalctl arguments. + """ + with tempfile.TemporaryDirectory() as tmp: + bin_dir = Path(tmp) + stub = bin_dir / "journalctl" + stub.write_text(JOURNALCTL_STUB) + stub.chmod(0o755) + (bin_dir / "journal").write_text( + "".join(json.dumps(entry) + "\n" for entry in entries) + ) + env = dict(os.environ) + env["PATH"] = f"{bin_dir}{os.pathsep}{env['PATH']}" + env["STUB_DIR"] = tmp + env["STUB_RC"] = str(rc) + done = subprocess.run( + [sys.executable, str(SCRIPT)], + env=env, + capture_output=True, + text=True, + check=False, + ) + args = (bin_dir / "args").read_text() # nocheck: cache-read tempdir fixture + return done.returncode, done.stdout, args + + def test_reads_the_journal_as_json_with_the_source_fields(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--output=json", args) + self.assertIn( + "--output-fields=MESSAGE,CONTAINER_ID,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT", + args, + ) + + def test_reads_every_field_in_full(self): + _, _, args = self.run_script(SYSTEM_INFO) + self.assertIn("--all", args.split()) + + def test_a_repeated_message_field_is_joined_and_reported(self): + code, output, _ = self.run_script(REPEATED_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn("zfs-import.service: unit started disk error", output) + self.assertIn('docker.service: level=error msg="failed to start', output) + + def test_an_application_error_from_a_container_is_ignored(self): + code, _, _ = self.run_script(APP_ERROR, SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_docker_daemon_error_is_reported_with_its_unit(self): + code, output, _ = self.run_script(APP_ERROR, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertIn( + 'docker.service: level=error msg="failed to start container"', output + ) + self.assertNotIn("pixelfed queue failed", output) + + def test_an_entry_without_unit_is_named_by_its_identifier(self): + code, output, _ = self.run_script(KERNEL_ERROR) + self.assertEqual(code, 1) + self.assertIn("kernel: EXT4-fs error (device sda1)", output) + + def test_a_non_utf8_message_is_decoded_and_reported(self): + code, output, _ = self.run_script(BINARY_ERROR) + self.assertEqual(code, 1) + self.assertIn("smartd.service: I/O error", output) + + def test_many_container_errors_are_streamed_past_and_not_reported(self): + container_lines = [ + {**APP_ERROR, "MESSAGE": f"ERROR: pixelfed request {n} failed"} + for n in range(20000) + ] + code, output, _ = self.run_script(*container_lines, DAEMON_ERROR) + self.assertEqual(code, 1) + self.assertNotIn("pixelfed request", output) + self.assertEqual( + output.splitlines()[-1], + 'docker.service: level=error msg="failed to start container"', + ) + self.assertEqual(len(output.splitlines()), 3) + + def test_a_journal_without_errors_passes(self): + code, _, _ = self.run_script(SYSTEM_INFO) + self.assertEqual(code, 0) + + def test_a_failing_journalctl_fails_the_check(self): + code, output, _ = self.run_script(rc=1) + self.assertEqual(code, 2) + self.assertIn("journalctl failed with exit code 1", output) + self.assertIn("unrecognized option '--output-fields'", output) + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/cache/README.md b/utils/cache/README.md index d62c5e0e2d..cfe7657783 100644 --- a/utils/cache/README.md +++ b/utils/cache/README.md @@ -26,7 +26,7 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read. -Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. +Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. ## Test-Only Helpers πŸ§ͺ