3 Commits

Author SHA1 Message Date
d115fc99b2 Release version 2.1.0 2026-08-22 17:26:20 +02:00
e96e8684e6 feat(i18n): translate labels, and keep brand names out of it
name and title were translated at render time but never machine-filled,
on the grounds that no backend tells the menu label "Pictures" from the
brand "Mastodon". That left the visible half of a card in English. They
are filled now, and the two key sets collapse into one.

The brands need somewhere to be named instead. app/i18n/keep.txt lists
them, one per line, and every entry is stored as itself in every target
language: no request, and never over an entry written by hand. --keep adds
one-off strings, --keep-file points elsewhere.

The shipped list holds the 43 product names that appear as name: or title:
in config.sample.yaml. Generic labels — Pictures, Imprint, Settings,
Certificates — are deliberately absent, and so are Cybermaster, Polymath
and Yachtmaster, which read as brand or as job title depending on who is
asking.

Two of these behaviours first shipped unguarded. A test that protected a
string and asserted the hand-written value survived passed either way,
because a run where nothing is missing reports "complete" and never
writes; and nothing exercised main(), so the keep file could stop being
read without a failure. Both are covered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:19:16 +02:00
747ce379cc fix(app): stop trusting X-Forwarded-For, and pin what the audit found
ProxyFix defaults x_for to 1, so ProxyFix(app.wsgi_app, x_proto=1) never
disabled it: request.remote_addr and the access log were forgeable by any
client that reached the app directly. It is x_for=0 now, asserted rather
than assumed.

A mutation audit over the change set reverted 196 deliberate behaviours
and found 47 that no test noticed. This closes the ones that carry damage:

- apod_background lost its key check, its transport guard, its status
  guard and its media-type check without a single test failing. Each one
  turns a slow or unhappy NASA into a 500 on every page.
- Untrusted values reached innerHTML through window.I18N, which the
  translation backend writes, and the modal's click handlers stacked so a
  later click opened an earlier popup's URL.
- The sync tool could ask for HTML instead of text, translate from "auto"
  instead of English, run without a timeout, store an empty translation
  that marks the string done for good, abandon 28 languages because one
  could not be written, and report success after reaching nothing.
- Neither the lint target, the CI jobs, the vendored RTL stylesheet, the
  documented environment keys, nor any of the four hardenings in
  scripts/run-e2e.sh was observed by anything.

Three of the new tests passed for the wrong reason on their first cut —
a mock that answered None whether or not the guard existed, a
raise_for_status that was never called, a string that stayed in the file
after the mutation. The audit found those too; all 24 reverts now fail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:19:15 +02:00
12 changed files with 534 additions and 29 deletions

View File

@@ -1,3 +1,16 @@
# Changelog
## [2.1.0] - 2026-08-22
* Multilingual site: every ISO 639-1 language has its own URL, */* follows the visitor's browser language, and a switcher in the navbar lists all 184 in their own script
* Right-to-left layouts: Arabic, Hebrew, Persian, Urdu and six more mirror the page automatically
* Machine translation: *make i18n* fills your card and menu texts from a LibreTranslate instance, *make i18n-ui* the interface strings; hand-corrected entries are never overwritten
* Brand protection: list your product names in *app/i18n/keep.txt* and they stay untranslated in every language
* Search engines: each page declares a canonical URL and an *hreflang* alternate per language
* Security: configuration and translations are HTML-escaped, script URLs are stripped from links, images and the iframe view — set *TRUSTED_HOSTS* in *.env* to pin your public hostname
* Reliability: a slow NASA APOD lookup no longer takes the page down
* Developer tooling: *make test-e2e* runs Cypress directly instead of through act, and YAML, JavaScript and shell now have linters in CI
## [2.0.0] - 2026-05-18 ## [2.0.0] - 2026-05-18
* * Asset resolution: new probe-first resolver tries a HEAD request and embeds reachable image URLs directly via a new external_url field, falling back to the cache-download path only when the probe fails; broken /static/https://... URLs no longer appear when the source cannot be downloaded * * Asset resolution: new probe-first resolver tries a HEAD request and embeds reachable image URLs directly via a new external_url field, falling back to the cache-download path only when the probe fails; broken /static/https://... URLs no longer appear when the source cannot be downloaded

View File

@@ -171,10 +171,25 @@ make i18n
This fills the interface strings of the languages that ship no catalogue as This fills the interface strings of the languages that ship no catalogue as
well. Existing entries are never overwritten, and a string the shipped well. Existing entries are never overwritten, and a string the shipped
catalogue already covers is never requested, so corrections you make by hand catalogue already covers is never requested, so corrections you make by hand
survive later runs. Only prose (`description`, `text`, `warning`, `info`, `subtitel`) is survive later runs.
filled automatically; `name` and `title` are left to you, because a machine
cannot tell the menu label "Pictures" from the brand "Mastodon". Write those `name`, `title`, `description`, `text`, `warning`, `info` and `subtitel` are
into the content catalogue yourself when you want them translated. translated; `url`, `link_text`, `identifier` and icon classes never are.
A machine cannot tell the menu label "Pictures" from the brand "Mastodon", so
list the brands in `app/i18n/keep.txt`, one per line — they are then stored as
themselves in every language and cost no request:
```
# Strings utils/i18n_sync.py stores as themselves instead of translating.
Mastodon
Nextcloud
freelancermap.de
```
Add one-off entries with `--keep Foo Bar`, or point somewhere else with
`--keep-file`. A protected string never replaces an entry you already wrote by
hand.
--- ---

View File

@@ -70,7 +70,7 @@ app = Flask(__name__)
app.jinja_options = {**app.jinja_options, "autoescape": True} app.jinja_options = {**app.jinja_options, "autoescape": True}
app.wsgi_app = ProxyFix(app.wsgi_app, x_proto=1) app.wsgi_app = ProxyFix(app.wsgi_app, x_for=0, x_proto=1)
def trusted_hosts(raw): def trusted_hosts(raw):

View File

@@ -72,6 +72,21 @@ describe('Untrusted content in the modal', () => {
cy.window().should('not.have.property', '__xss'); cy.window().should('not.have.property', '__xss');
}); });
it('keeps a relative link', () => {
open({ warning: 'See [the notes](/#anchor)' });
cy.get('#dynamicModalWarningText')
.find('a')
.should('have.attr', 'href', '/#anchor');
});
it('keeps the text of a link it strips', () => {
open({ warning: '[read this](javascript:window.__xss=1)' });
cy.get('#dynamicModalWarningText').should('contain.text', 'read this');
cy.window().should('not.have.property', '__xss');
});
it('keeps ordinary markdown', () => { it('keeps ordinary markdown', () => {
open({ warning: 'See [Matrix](https://matrix.org/) and **mind** this' }); open({ warning: 'See [Matrix](https://matrix.org/) and **mind** this' });
@@ -83,6 +98,40 @@ describe('Untrusted content in the modal', () => {
}); });
describe('values interpolated outside markdown', () => { describe('values interpolated outside markdown', () => {
it('does not treat an interface string as markup', () => {
cy.window().then(win => {
win.I18N.Open = '<img src=x onerror="window.__xss = true">';
});
open({
alternatives: [
{ name: 'Alt', identifier: 'A', icon: { class: 'fa-alt' } },
],
});
cy.get('#dynamicAlternativesList').find('img').should('not.exist');
cy.get('#dynamicAlternativesList').should('contain.text', 'onerror');
cy.window().should('not.have.property', '__xss');
});
it('falls back to the English source when a string is missing', () => {
cy.window().then(win => {
delete win.I18N;
});
open({
alternatives: [
{ name: 'Alt', identifier: 'A', icon: { class: 'fa-alt' } },
],
});
cy.get('#dynamicAlternativesList button').should('have.text', 'Open');
});
it('renders no placeholder for a missing name', () => {
open({ name: undefined });
cy.get('#dynamicModalLabel').should('not.contain.text', 'undefined');
});
it('does not treat the name or the icon class as markup', () => { it('does not treat the name or the icon class as markup', () => {
open({ open({
name: '<img src=x onerror="window.__xss = true">', name: '<img src=x onerror="window.__xss = true">',
@@ -123,6 +172,12 @@ describe('Untrusted content in the modal', () => {
); );
}); });
it('keeps a URL that carries surrounding whitespace', () => {
open({ url: ' https://example.com ', description: 'Good' });
cy.get('#dynamicModalLinkHref').should('have.attr', 'href');
});
it('keeps a mailto URL', () => { it('keeps a mailto URL', () => {
open({ url: 'mailto:kevin@veen.world', description: 'Write' }); open({ url: 'mailto:kevin@veen.world', description: 'Write' });
@@ -156,6 +211,17 @@ describe('Untrusted content in the modal', () => {
expect($anchor[0].onclick, 'stale click handler').to.equal(null); expect($anchor[0].onclick, 'stale click handler').to.equal(null);
}); });
}); });
it('opens the current popup URL, not an earlier one', () => {
open({ url: 'https://a.test/', description: 'A', iframe: true });
open({ url: 'https://b.test/', description: 'B', iframe: true });
cy.get('#dynamicModalLinkHref').click();
cy.get('#main')
.find('iframe', { timeout: 4000 })
.should('have.attr', 'src', 'https://b.test/');
});
}); });
}); });

46
app/i18n/keep.txt Normal file
View File

@@ -0,0 +1,46 @@
# Strings utils/i18n_sync.py stores as themselves instead of translating.
# One per line; blank lines and lines starting with # are ignored.
Baserow
Big Blue Button
Bluesky
Buy me a Coffee
Discourse
Duolingo
Eversports
Facebook
Friendica
Garmin
Gitea
GitHub
GitHub Sponsors
Infinito.Nexus
Instagram
Keycloak
LDAP
LinkedIn
Mailu
Mastodon
Matomo
Matrix
Nextcloud
Open Project
Patreon
PayPal
Peertube
Pixelfed
Signal
Snipe IT
Spotify
Taiga
Telegram
Twitter
WhatsApp
XING
YouTube
Yourls
Zoom
freelancermap.de
malt
phpMyAdmin
upwork.com

View File

@@ -23,9 +23,9 @@ CONTENT_DIR = I18N_DIR / "content"
SOURCE_LANGUAGE = "en" SOURCE_LANGUAGE = "en"
AUTOFILL_KEYS = frozenset({"description", "text", "warning", "info", "subtitel"}) TRANSLATABLE_KEYS = frozenset(
{"description", "info", "name", "subtitel", "text", "title", "warning"}
TRANSLATABLE_KEYS = AUTOFILL_KEYS | frozenset({"name", "title"}) )
UI_STRINGS = ( UI_STRINGS = (
"Alternatives", "Alternatives",

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "portfolio-ui" name = "portfolio-ui"
version = "2.0.0" version = "2.1.0"
description = "A lightweight YAML-driven portfolio and landing-page generator." description = "A lightweight YAML-driven portfolio and landing-page generator."
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"

View File

@@ -6,6 +6,9 @@ import sys
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest.mock import Mock, patch
import requests
REPO_ROOT = Path(__file__).resolve().parents[2] REPO_ROOT = Path(__file__).resolve().parents[2]
@@ -92,15 +95,75 @@ class TestNegotiation(AppRouteMixin, unittest.TestCase):
class TestEscaping(AppRouteMixin, unittest.TestCase): class TestEscaping(AppRouteMixin, unittest.TestCase):
def test_catalog_content_is_html_escaped(self): def test_catalog_content_is_html_escaped(self):
i18n._catalogs["de"] = {"Imprint": "<script>alert(1)</script>"} i18n._catalogs["de"] = {"Copy": "<script>alert('ui')</script>"}
body = self.client.get("/de/").get_data(as_text=True) body = self.client.get("/de/").get_data(as_text=True)
self.assertNotIn("<script>alert(1)</script>", body) self.assertNotIn("<script>alert('ui')</script>", body)
self.assertIn("&lt;script&gt;alert(1)", body) self.assertIn("&lt;script&gt;alert(&#39;ui&#39;)", body)
def test_configuration_content_is_html_escaped(self):
i18n._catalogs["de"] = {"Imprint": "<script>alert('config')</script>"}
body = self.client.get("/de/").get_data(as_text=True)
self.assertNotIn("<script>alert('config')</script>", body)
self.assertIn("&lt;script&gt;alert(&#39;config&#39;)", body)
class TestApodBackground(AppRouteMixin, unittest.TestCase):
def setUp(self):
super().setUp()
self.addCleanup(flask_app.config.__setitem__, "NASA_API_KEY", None)
flask_app.config["NASA_API_KEY"] = "key"
def test_no_request_is_made_without_a_key(self):
flask_app.config["NASA_API_KEY"] = None
with patch("app.app.requests.get") as get:
self.assertIsNone(app_module.apod_background())
get.assert_not_called()
def test_a_transport_failure_costs_the_background_not_the_page(self):
with patch(
"app.app.requests.get", side_effect=requests.ConnectionError("down")
):
self.assertIsNone(app_module.apod_background())
self.assertEqual(self.client.get("/en/").status_code, 200)
def test_an_error_response_costs_the_background_not_the_page(self):
refusal = Mock(ok=False)
refusal.json.return_value = {"media_type": "image", "url": "https://i.test/x"}
with patch("app.app.requests.get", return_value=refusal):
self.assertIsNone(app_module.apod_background())
def test_a_video_of_the_day_is_not_used_as_a_background(self):
answer = Mock(ok=True)
answer.json.return_value = {"media_type": "video", "url": "https://v.test/x"}
with patch("app.app.requests.get", return_value=answer):
self.assertIsNone(app_module.apod_background())
def test_an_image_of_the_day_is_used(self):
answer = Mock(ok=True)
answer.json.return_value = {"media_type": "image", "url": "https://i.test/x"}
with patch("app.app.requests.get", return_value=answer):
self.assertEqual(app_module.apod_background(), "https://i.test/x")
class TestExternalUrls(AppRouteMixin, unittest.TestCase): class TestExternalUrls(AppRouteMixin, unittest.TestCase):
def test_only_the_forwarded_scheme_is_trusted(self):
proxy = flask_app.wsgi_app
self.assertEqual(proxy.x_proto, 1)
self.assertEqual(
(proxy.x_for, proxy.x_host, proxy.x_port, proxy.x_prefix), (0, 0, 0, 0)
)
def test_forwarded_scheme_reaches_the_canonical_and_alternates(self): def test_forwarded_scheme_reaches_the_canonical_and_alternates(self):
body = self.client.get( body = self.client.get(
"/en/", "/en/",

View File

@@ -5,6 +5,7 @@ checks: the deletion is invisible to every suite, and its effect only shows up
in production or in a fresh checkout. in production or in a fresh checkout.
""" """
import json
import re import re
import tomllib import tomllib
import unittest import unittest
@@ -64,6 +65,84 @@ class TestRunTargets(unittest.TestCase):
self.assertNotIn("--env-file", body) self.assertNotIn("--env-file", body)
class TestLintCoverage(unittest.TestCase):
def setUp(self):
self.makefile = (REPO_ROOT / "Makefile").read_text(encoding="utf-8")
def test_the_lint_target_runs_every_linter(self):
prerequisites = re.search(r"^lint: (.+)$", self.makefile, re.MULTILINE).group(1)
self.assertGreaterEqual(
set(prerequisites.split()),
{"lint-actions", "lint-python", "lint-yaml", "lint-js", "lint-shell"},
)
def test_every_linter_has_a_ci_job(self):
workflow = yaml.safe_load(
(REPO_ROOT / ".github" / "workflows" / "lint.yml").read_text(
encoding="utf-8"
)
)
self.assertGreaterEqual(
set(workflow["jobs"]),
{"lint-actions", "lint-python", "lint-yaml", "lint-js", "lint-shell"},
)
def test_the_javascript_linter_is_declared(self):
package = json.loads(
(REPO_ROOT / "app" / "package.json").read_text(encoding="utf-8")
)
self.assertGreaterEqual(
set(package["devDependencies"]), {"eslint", "@eslint/js", "globals"}
)
def test_the_documented_environment_keys_exist(self):
example = (REPO_ROOT / "env.example").read_text(encoding="utf-8")
for key in ("PORT", "IMAGE_NAME", "TRUSTED_HOSTS", "LIBRETRANSLATE_URL"):
with self.subTest(key=key):
self.assertRegex(example, rf"(?m)^{key}=")
class TestEndToEndRunner(unittest.TestCase):
def setUp(self):
self.script = (REPO_ROOT / "scripts" / "run-e2e.sh").read_text(encoding="utf-8")
def test_a_foreign_listener_stops_the_run(self):
self.assertIn("already serves port", self.script)
def test_cypress_is_pinned_to_the_origin_flask_binds(self):
self.assertIn("CYPRESS_baseUrl", self.script)
self.assertIn("127.0.0.1", self.script)
def test_the_electron_node_flag_is_dropped(self):
self.assertIn("env -u ELECTRON_RUN_AS_NODE", self.script)
def test_every_probe_bypasses_a_proxy_and_is_bounded(self):
probes = [line for line in self.script.splitlines() if "curl " in line]
self.assertTrue(probes)
for probe in probes:
with self.subTest(probe=probe.strip()):
self.assertIn("--noproxy", probe)
self.assertIn("--max-time", probe)
class TestVendoredAssets(unittest.TestCase):
def test_the_right_to_left_stylesheet_is_vendored(self):
script = (REPO_ROOT / "app" / "scripts" / "copy-vendor.js").read_text(
encoding="utf-8"
)
self.assertEqual(
script.count("bootstrap.rtl.min.css"),
2,
"the RTL stylesheet needs both a source and a destination path",
)
class TestPackagedCatalogs(unittest.TestCase): class TestPackagedCatalogs(unittest.TestCase):
def test_the_interface_catalogs_are_declared_as_package_data(self): def test_the_interface_catalogs_are_declared_as_package_data(self):
with (REPO_ROOT / "pyproject.toml").open("rb") as handle: with (REPO_ROOT / "pyproject.toml").open("rb") as handle:

View File

@@ -13,6 +13,9 @@ class TestNegotiate(unittest.TestCase):
def test_regional_tag_beats_a_lower_ranked_exact_match(self): def test_regional_tag_beats_a_lower_ranked_exact_match(self):
self.assertEqual(i18n.negotiate([("de-DE", 1.0), ("en", 0.8)]), "de") self.assertEqual(i18n.negotiate([("de-DE", 1.0), ("en", 0.8)]), "de")
def test_an_uppercase_tag_is_accepted(self):
self.assertEqual(i18n.negotiate([("DE-DE", 1.0)]), "de")
def test_underscore_separated_tag_is_accepted(self): def test_underscore_separated_tag_is_accepted(self):
self.assertEqual(i18n.negotiate([("pt_BR", 1.0)]), "pt") self.assertEqual(i18n.negotiate([("pt_BR", 1.0)]), "pt")
@@ -70,6 +73,11 @@ class TestTranslateTree(unittest.TestCase):
self.assertEqual(card["url"], "A card") self.assertEqual(card["url"], "A card")
self.assertEqual(card["icon"]["class"], "Pictures") self.assertEqual(card["icon"]["class"], "Pictures")
def test_strings_inside_a_list_are_translated(self):
translated = i18n.translate_tree({"text": ["A card", "Pictures"]}, "xx")
self.assertEqual(translated["text"], ["Eine Karte", "Bilder"])
def test_unknown_strings_keep_their_source_value(self): def test_unknown_strings_keep_their_source_value(self):
translated = i18n.translate_tree({"description": "Untranslated"}, "xx") translated = i18n.translate_tree({"description": "Untranslated"}, "xx")
@@ -125,11 +133,16 @@ class TestReadCatalog(unittest.TestCase):
def test_non_string_entries_are_dropped(self): def test_non_string_entries_are_dropped(self):
self.path.write_text( self.path.write_text(
"Close: 42\nOpen:\nCopy: yes\nImprint: Impressum\n", encoding="utf-8" "Close: 42\nOpen:\nCopy: yes\n123: Zahl\nyes: Ja\nImprint: Impressum\n",
encoding="utf-8",
) )
self.assertEqual(i18n.read_catalog(self.path), {"Imprint": "Impressum"}) self.assertEqual(i18n.read_catalog(self.path), {"Imprint": "Impressum"})
def test_a_missing_catalog_is_silent(self):
with self.assertNoLogs(level="WARNING"):
i18n.read_catalog(self.directory / "absent.yaml")
class TestCatalogMerge(unittest.TestCase): class TestCatalogMerge(unittest.TestCase):
def setUp(self): def setUp(self):

View File

@@ -30,19 +30,26 @@ class TestCollectSources(unittest.TestCase):
self.assertEqual( self.assertEqual(
i18n_sync.collect_sources(config), i18n_sync.collect_sources(config),
{ {
"Agile Coach",
"I lead transformations.", "I lead transformations.",
"Another card.", "Another card.",
"Apps",
"Application menu", "Application menu",
"A tagline", "A tagline",
}, },
) )
def test_label_and_structural_keys_are_left_alone(self): def test_labels_are_collected_too(self):
config = {"name": "Pictures", "title": "Agile Coach"}
self.assertEqual(i18n_sync.collect_sources(config), {"Pictures", "Agile Coach"})
def test_structural_keys_are_left_alone(self):
config = { config = {
"name": "Mastodon",
"title": "Cybermaster",
"url": "https://example.test", "url": "https://example.test",
"link_text": "www.example.test", "link_text": "www.example.test",
"identifier": "@someone@example.test",
"icon": {"class": "fa-solid fa-users"},
} }
self.assertEqual(i18n_sync.collect_sources(config), set()) self.assertEqual(i18n_sync.collect_sources(config), set())
@@ -50,6 +57,11 @@ class TestCollectSources(unittest.TestCase):
def test_blank_values_are_ignored(self): def test_blank_values_are_ignored(self):
self.assertEqual(i18n_sync.collect_sources({"text": " "}), set()) self.assertEqual(i18n_sync.collect_sources({"text": " "}), set())
def test_a_list_of_prose_is_collected(self):
self.assertEqual(
i18n_sync.collect_sources({"text": ["one", "two"]}), {"one", "two"}
)
class TestTranslate(unittest.TestCase): class TestTranslate(unittest.TestCase):
def _session(self, ok, payload=None, status=200, text=""): def _session(self, ok, payload=None, status=200, text=""):
@@ -79,7 +91,33 @@ class TestTranslate(unittest.TestCase):
self.assertNotIn("api_key", session.post.call_args.kwargs["data"]) self.assertNotIn("api_key", session.post.call_args.kwargs["data"])
def test_failed_response_returns_none(self): def test_failed_response_returns_none(self):
session = self._session(False, status=403, text="denied") session = self._session(
False, {"translatedText": "SHOULD NOT BE USED"}, status=403, text="denied"
)
self.assertIsNone(i18n_sync.translate(session, "http://lt", "", "Hi", "de"))
def test_the_request_asks_for_plain_text_from_the_source_language(self):
session = self._session(True, {"translatedText": "Hallo"})
i18n_sync.translate(session, "http://lt", "", "Hello", "de")
data = session.post.call_args.kwargs["data"]
self.assertEqual(data["format"], "text")
self.assertEqual(data["source"], i18n.SOURCE_LANGUAGE)
self.assertEqual(data["target"], "de")
def test_the_request_is_bounded_by_a_timeout(self):
session = self._session(True, {"translatedText": "Hallo"})
i18n_sync.translate(session, "http://lt", "", "Hello", "de")
timeout = session.post.call_args.kwargs["timeout"]
self.assertIsInstance(timeout, (int, float))
self.assertGreater(timeout, 0)
def test_an_empty_translation_is_refused(self):
session = self._session(True, {"translatedText": ""})
self.assertIsNone(i18n_sync.translate(session, "http://lt", "", "Hi", "de")) self.assertIsNone(i18n_sync.translate(session, "http://lt", "", "Hi", "de"))
@@ -125,6 +163,15 @@ class TestSupportedLanguages(unittest.TestCase):
with self.assertRaises(i18n_sync.BackendError): with self.assertRaises(i18n_sync.BackendError):
i18n_sync.supported_languages("http://lt", session) i18n_sync.supported_languages("http://lt", session)
def test_an_error_status_is_reported(self):
response = Mock()
response.raise_for_status.side_effect = i18n_sync.requests.HTTPError("503")
response.json.return_value = [{"code": "de"}]
session = Mock(get=Mock(return_value=response))
with self.assertRaises(i18n_sync.BackendError):
i18n_sync.supported_languages("http://lt", session)
def test_a_non_json_listing_is_reported(self): def test_a_non_json_listing_is_reported(self):
session = self._session(side_effect=ValueError("no json")) session = self._session(side_effect=ValueError("no json"))
@@ -138,6 +185,80 @@ class TestSupportedLanguages(unittest.TestCase):
i18n_sync.supported_languages("http://lt", self._session(payload)) i18n_sync.supported_languages("http://lt", self._session(payload))
class TestReadKeep(unittest.TestCase):
def setUp(self):
self.directory = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.directory, True)
self.path = self.directory / "keep.txt"
def test_a_missing_file_keeps_nothing(self):
self.assertEqual(i18n_sync.read_keep(self.path), [])
def test_comments_and_blank_lines_are_ignored(self):
self.path.write_text(
"# brands\n\nMastodon\n Bluesky \n\n# more\nNextcloud\n",
encoding="utf-8",
)
self.assertEqual(
i18n_sync.read_keep(self.path), ["Mastodon", "Bluesky", "Nextcloud"]
)
def test_the_shipped_list_names_the_brands_of_the_sample_configuration(self):
shipped = i18n_sync.read_keep(i18n_sync.DEFAULT_KEEP_PATH)
self.assertGreaterEqual(set(shipped), {"Mastodon", "Nextcloud", "Matrix"})
self.assertNotIn("Pictures", shipped)
self.assertNotIn("Imprint", shipped)
class TestCommandLine(unittest.TestCase):
def setUp(self):
self.directory = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.directory, True)
self.keep_file = self.directory / "keep.txt"
self.config = self.directory / "config.yaml"
self.config.write_text("cards: []\n", encoding="utf-8")
def _main(self, *extra):
with patch.object(i18n_sync, "sync") as sync:
i18n_sync.main(
[
"--url",
"http://lt/",
"--config",
str(self.config),
"--keep-file",
str(self.keep_file),
*extra,
]
)
return sync.call_args
def test_the_keep_file_reaches_the_sync(self):
self.keep_file.write_text("# brands\nMastodon\nNextcloud\n", encoding="utf-8")
keep = self._main().args[5]
self.assertEqual(list(keep), ["Mastodon", "Nextcloud"])
def test_the_command_line_adds_to_the_keep_file(self):
self.keep_file.write_text("Mastodon\n", encoding="utf-8")
keep = self._main("--keep", "Taiga").args[5]
self.assertEqual(list(keep), ["Mastodon", "Taiga"])
def test_a_trailing_slash_is_stripped_from_the_url(self):
self.assertEqual(self._main().args[0], "http://lt")
def test_a_backend_failure_is_reported_as_an_exit_code(self):
with patch.object(i18n_sync, "sync", side_effect=i18n_sync.BackendError("no")):
self.assertEqual(
i18n_sync.main(["--url", "http://lt", "--config", str(self.config)]), 1
)
class TestLoadExisting(unittest.TestCase): class TestLoadExisting(unittest.TestCase):
def setUp(self): def setUp(self):
self.directory = Path(tempfile.mkdtemp()) self.directory = Path(tempfile.mkdtemp())
@@ -186,11 +307,11 @@ class TestWriteCatalog(unittest.TestCase):
self.path = self.directory / "de.yaml" self.path = self.directory / "de.yaml"
def test_the_catalog_is_written_and_no_scratch_file_is_left(self): def test_the_catalog_is_written_and_no_scratch_file_is_left(self):
i18n_sync.write_catalog(self.path, {"Hi": "Hallo"}) i18n_sync.write_catalog(self.path, {"Hi": "Hallo", "Umlaut": "Grüße"})
self.assertEqual( text = self.path.read_text(encoding="utf-8")
yaml.safe_load(self.path.read_text(encoding="utf-8")), {"Hi": "Hallo"} self.assertEqual(yaml.safe_load(text), {"Hi": "Hallo", "Umlaut": "Grüße"})
) self.assertIn("Grüße", text)
self.assertEqual([p.name for p in self.directory.iterdir()], ["de.yaml"]) self.assertEqual([p.name for p in self.directory.iterdir()], ["de.yaml"])
def test_a_write_that_dies_halfway_leaves_the_previous_catalog_intact(self): def test_a_write_that_dies_halfway_leaves_the_previous_catalog_intact(self):
@@ -216,7 +337,7 @@ class TestSync(unittest.TestCase):
i18n.CONTENT_DIR = self.directory i18n.CONTENT_DIR = self.directory
self.path = self.directory / "de.yaml" self.path = self.directory / "de.yaml"
def _run(self, sources, offered=("de",), translated="UEBERSETZT"): def _run(self, sources, offered=("de",), translated="UEBERSETZT", keep=()):
listing = Mock(raise_for_status=Mock()) listing = Mock(raise_for_status=Mock())
listing.json.return_value = [{"code": code} for code in offered] listing.json.return_value = [{"code": code} for code in offered]
answer = Mock(ok=True) answer = Mock(ok=True)
@@ -224,7 +345,7 @@ class TestSync(unittest.TestCase):
session = Mock(get=Mock(return_value=listing), post=Mock(return_value=answer)) session = Mock(get=Mock(return_value=listing), post=Mock(return_value=answer))
with patch.object(i18n_sync.requests, "Session", return_value=session): with patch.object(i18n_sync.requests, "Session", return_value=session):
i18n_sync.sync("http://lt", "", set(sources), ["de"], self.directory) i18n_sync.sync("http://lt", "", set(sources), ["de"], self.directory, keep)
return session return session
def test_missing_entries_are_written(self): def test_missing_entries_are_written(self):
@@ -259,6 +380,27 @@ class TestSync(unittest.TestCase):
self.assertFalse(self.path.exists()) self.assertFalse(self.path.exists())
session.post.assert_not_called() session.post.assert_not_called()
def test_a_protected_string_is_stored_as_itself(self):
session = self._run(["Mastodon", "Hello"], keep=["Mastodon"])
catalog = yaml.safe_load(self.path.read_text(encoding="utf-8"))
self.assertEqual(catalog["Mastodon"], "Mastodon")
self.assertEqual(session.post.call_count, 1)
def test_a_protected_string_never_overwrites_an_existing_entry(self):
self.path.write_text("Mastodon: HAND-EDITED\n", encoding="utf-8")
self._run(["Mastodon", "Hello"], keep=["Mastodon"])
catalog = yaml.safe_load(self.path.read_text(encoding="utf-8"))
self.assertEqual(catalog["Mastodon"], "HAND-EDITED")
self.assertEqual(catalog["Hello"], "UEBERSETZT")
def test_an_empty_translation_is_not_stored(self):
self._run(["Hello"], translated="")
self.assertFalse(self.path.exists())
def test_a_run_that_translated_nothing_leaves_the_file_untouched(self): def test_a_run_that_translated_nothing_leaves_the_file_untouched(self):
original = "# Reviewed by a native speaker, keep the order.\nZebra: Zebra\n" original = "# Reviewed by a native speaker, keep the order.\nZebra: Zebra\n"
self.path.write_text(original, encoding="utf-8") self.path.write_text(original, encoding="utf-8")
@@ -279,6 +421,25 @@ class TestSync(unittest.TestCase):
self.assertEqual(session.post.call_count, 1) self.assertEqual(session.post.call_count, 1)
self.assertNotIn("Close", yaml.safe_load(self.path.read_text(encoding="utf-8"))) self.assertNotIn("Close", yaml.safe_load(self.path.read_text(encoding="utf-8")))
def test_a_language_that_cannot_be_written_does_not_stop_the_others(self):
listing = Mock(raise_for_status=Mock())
listing.json.return_value = [{"code": "de"}, {"code": "fr"}]
answer = Mock(ok=True)
answer.json.return_value = {"translatedText": "UEBERSETZT"}
session = Mock(get=Mock(return_value=listing), post=Mock(return_value=answer))
original = i18n_sync.write_catalog
def fails_for_german(path, catalog):
if path.name == "de.yaml":
raise OSError("read-only")
original(path, catalog)
with patch.object(i18n_sync, "write_catalog", fails_for_german):
with patch.object(i18n_sync.requests, "Session", return_value=session):
i18n_sync.sync("http://lt", "", {"Hello"}, ["de", "fr"], self.directory)
self.assertTrue((self.directory / "fr.yaml").is_file())
def test_the_catalog_directory_is_created(self): def test_the_catalog_directory_is_created(self):
nested = self.directory / "content" nested = self.directory / "content"
self.directory = nested self.directory = nested

View File

@@ -20,6 +20,7 @@ sys.path.insert(0, str(REPO_ROOT))
from app.utils import i18n # noqa: E402 from app.utils import i18n # noqa: E402
DEFAULT_CONFIG_PATH = REPO_ROOT / "app" / "config.yaml" DEFAULT_CONFIG_PATH = REPO_ROOT / "app" / "config.yaml"
DEFAULT_KEEP_PATH = i18n.I18N_DIR / "keep.txt"
REQUEST_TIMEOUT = 30 REQUEST_TIMEOUT = 30
@@ -38,11 +39,27 @@ def collect_sources(node, key=None, found=None):
elif isinstance(node, list): elif isinstance(node, list):
for item in node: for item in node:
collect_sources(item, key, found) collect_sources(item, key, found)
elif isinstance(node, str) and key in i18n.AUTOFILL_KEYS and node.strip(): elif isinstance(node, str) and key in i18n.TRANSLATABLE_KEYS and node.strip():
found.add(node) found.add(node)
return found return found
def read_keep(path):
"""Return the strings listed in ``path``, ignoring blanks and comments.
Args:
path: a text file with one string per line, or a path that does not exist.
"""
if not path.is_file():
return []
lines = path.read_text(encoding="utf-8").splitlines()
return [
stripped
for stripped in (line.strip() for line in lines)
if stripped and not stripped.startswith("#")
]
class BackendError(Exception): class BackendError(Exception):
"""The translation backend answered in a way the run cannot continue from.""" """The translation backend answered in a way the run cannot continue from."""
@@ -138,13 +155,13 @@ def translate(session, url, api_key, text, target):
print(f" ! {target}: answered 200 but not JSON") print(f" ! {target}: answered 200 but not JSON")
return None return None
if not isinstance(translated, str): if not isinstance(translated, str) or not translated.strip():
print(f" ! {target}: translatedText was {type(translated).__name__}") print(f" ! {target}: unusable translatedText ({translated!r})")
return None return None
return translated return translated
def sync(url, api_key, sources, targets, directory): def sync(url, api_key, sources, targets, directory, keep=()):
"""Fill and write the catalogue of every language in ``targets``. """Fill and write the catalogue of every language in ``targets``.
Args: Args:
@@ -153,6 +170,7 @@ def sync(url, api_key, sources, targets, directory):
sources: English strings to translate. sources: English strings to translate.
targets: language codes to fill. targets: language codes to fill.
directory: catalogue directory to write into. directory: catalogue directory to write into.
keep: strings to store as themselves instead of translating.
""" """
directory.mkdir(parents=True, exist_ok=True) directory.mkdir(parents=True, exist_ok=True)
session = requests.Session() session = requests.Session()
@@ -168,6 +186,9 @@ def sync(url, api_key, sources, targets, directory):
if catalog is None: if catalog is None:
continue continue
protected = {word: word for word in keep if word not in catalog}
catalog.update(protected)
shipped = ( shipped = (
{} {}
if directory == i18n.UI_DIR if directory == i18n.UI_DIR
@@ -190,7 +211,7 @@ def sync(url, api_key, sources, targets, directory):
catalog[source] = translated catalog[source] = translated
added += 1 added += 1
if not added: if not added and not protected:
print(f" ! {target}: nothing translated, leaving the file untouched") print(f" ! {target}: nothing translated, leaving the file untouched")
continue continue
@@ -227,6 +248,24 @@ def main(argv=None):
"ui: the interface strings that ship with the project." "ui: the interface strings that ship with the project."
), ),
) )
parser.add_argument(
"--keep",
nargs="+",
default=[],
metavar="STRING",
help=(
"Store these as themselves instead of translating, "
"in addition to --keep-file."
),
)
parser.add_argument(
"--keep-file",
type=Path,
default=DEFAULT_KEEP_PATH,
help=(
f"One string per line, stored untranslated (default: {DEFAULT_KEEP_PATH})."
),
)
parser.add_argument( parser.add_argument(
"--languages", "--languages",
nargs="+", nargs="+",
@@ -245,8 +284,18 @@ def main(argv=None):
sources = collect_sources(config) | set(i18n.UI_STRINGS) sources = collect_sources(config) | set(i18n.UI_STRINGS)
print(f"{len(sources)} string(s) in {args.config} and the interface") print(f"{len(sources)} string(s) in {args.config} and the interface")
keep = read_keep(args.keep_file) + args.keep
print(f"{len(keep)} string(s) kept untranslated")
try: try:
sync(args.url.rstrip("/"), args.api_key, sources, args.languages, directory) sync(
args.url.rstrip("/"),
args.api_key,
sources,
args.languages,
directory,
keep,
)
except BackendError as error: except BackendError as error:
print(f"ERROR: {error}", file=sys.stderr) print(f"ERROR: {error}", file=sys.stderr)
return 1 return 1