From 632dce2d0e09f972af51f0fadb1d98ee3e6f3cf4 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Mon, 21 Apr 2025 11:01:04 -0700 Subject: [PATCH 1/7] Fix unpaired surrogate in test Fixes #1062 per 2025-04-21 call --- test/tests/syntax-errors.json | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/test/tests/syntax-errors.json b/test/tests/syntax-errors.json index 12f41826a4..b0861eda7c 100644 --- a/test/tests/syntax-errors.json +++ b/test/tests/syntax-errors.json @@ -192,7 +192,11 @@ { "src": "{^.}" }, { "src": "{^ .}" }, { "src": "{&}" }, - { "src": "{\ud800}" }, +// { "src": "{\ud800}" }, Unpaired surrogate test. +// This is not valid JSON and some JSON parsers will not parse this test. +// Uncomment if your implementation supports unpaired surrogates +// for text and wants to test invalid placeholders. +// See: https://www.unicode.org/reports/tr35/dev/tr35-messageFormat.html#the-message { "src": "{\ufdd0}" }, { "src": "{\ufffe}" }, { "src": "{!.\\{}" }, From 0d3d4673cfa8e84677e883ddb8c87464fd00dcf0 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Mon, 21 Apr 2025 11:54:05 -0700 Subject: [PATCH 2/7] Remove commenting Comments, of course, are not legal JSON either, but sometimes work... removing here so we can handle differently. --- test/tests/syntax-errors.json | 5 ----- 1 file changed, 5 deletions(-) diff --git a/test/tests/syntax-errors.json b/test/tests/syntax-errors.json index b0861eda7c..7f840b3cf4 100644 --- a/test/tests/syntax-errors.json +++ b/test/tests/syntax-errors.json @@ -192,11 +192,6 @@ { "src": "{^.}" }, { "src": "{^ .}" }, { "src": "{&}" }, -// { "src": "{\ud800}" }, Unpaired surrogate test. -// This is not valid JSON and some JSON parsers will not parse this test. -// Uncomment if your implementation supports unpaired surrogates -// for text and wants to test invalid placeholders. -// See: https://www.unicode.org/reports/tr35/dev/tr35-messageFormat.html#the-message { "src": "{\ufdd0}" }, { "src": "{\ufffe}" }, { "src": "{!.\\{}" }, From 9470b46fb631acc0922abdf79558f87c03c12c14 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Mon, 21 Apr 2025 12:03:36 -0700 Subject: [PATCH 3/7] Update syntax-errors.json --- test/tests/syntax-errors.json | 3 +++ 1 file changed, 3 insertions(+) diff --git a/test/tests/syntax-errors.json b/test/tests/syntax-errors.json index 7f840b3cf4..516fa823bb 100644 --- a/test/tests/syntax-errors.json +++ b/test/tests/syntax-errors.json @@ -192,6 +192,9 @@ { "src": "{^.}" }, { "src": "{^ .}" }, { "src": "{&}" }, + { "description": "Valid JSON cannot contain an unpaired surrogate. Unescape backslash to create a syntax error for implementations that support UTF-16.", + "src": "{\\ud800}" + }, { "src": "{\ufdd0}" }, { "src": "{\ufffe}" }, { "src": "{!.\\{}" }, From 7796e1062e07723542e82b871b0f08f9af241988 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Tue, 22 Apr 2025 08:06:31 -0700 Subject: [PATCH 4/7] Update test/tests/syntax-errors.json Co-authored-by: Eemeli Aro --- test/tests/syntax-errors.json | 3 --- 1 file changed, 3 deletions(-) diff --git a/test/tests/syntax-errors.json b/test/tests/syntax-errors.json index 516fa823bb..7f840b3cf4 100644 --- a/test/tests/syntax-errors.json +++ b/test/tests/syntax-errors.json @@ -192,9 +192,6 @@ { "src": "{^.}" }, { "src": "{^ .}" }, { "src": "{&}" }, - { "description": "Valid JSON cannot contain an unpaired surrogate. Unescape backslash to create a syntax error for implementations that support UTF-16.", - "src": "{\\ud800}" - }, { "src": "{\ufdd0}" }, { "src": "{\ufffe}" }, { "src": "{!.\\{}" }, From 22b641967f5c6662ba7d3a3908070a4093ccb26f Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Tue, 22 Apr 2025 08:17:42 -0700 Subject: [PATCH 5/7] Update README.md --- test/README.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/test/README.md b/test/README.md index de39fdffa7..f046147902 100644 --- a/test/README.md +++ b/test/README.md @@ -1,12 +1,21 @@ -The tests in the `./tests/` directory were originally copied from the [messageformat project](https://github.com/messageformat/messageformat/tree/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/__fixtures) -and are here relicensed by their original author (Eemeli Aro) under the Unicode License. +# Unicode MessageFormat Test Suite -These test files are intended to be useful for testing multiple different message processors in different ways: +These test files are intended to be useful for testing multiple different _message_ processors in different ways: - `syntax.json` — Test cases that do not depend on any registry definitions. - `syntax-errors.json` — Strings that should produce a Syntax Error when parsed. +> [!NOTE] +> `syntax-errors.json` does not include tests for the disallowed uses of unpaired surrogate code points +> because JSON does not permit unpaired surrogate code points. +> If your implementation uses UTF-16 based strings (such as JavaScript `String` or Java `java.lang.String`) +> or otherwise allows unpaired surrogates in text or literals, you will need to implement tests equivalent +> to the following for syntax errors: +> ```json +> { "src": "{\ud800}" } +> ``` + - `data-model-errors.json` - Strings that should produce a Data Model Error when processed. Error names are defined in ["MessageFormat 2.0 Errors"](../spec/errors.md) in the spec. @@ -192,3 +201,8 @@ except that it cannot be used for selection. When `:test:format` is used as a _selector_, the steps under 2.iii. of [Resolve Selectors](/spec/formatting.md#resolve-selectors) are followed. + +## About + +The tests in the `./tests/` directory were originally copied from the [messageformat project](https://github.com/messageformat/messageformat/tree/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/__fixtures) +and are here relicensed by their original author (Eemeli Aro) under the Unicode License. From e5ef8add4f215379fb065fc041cf223df1a456c8 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Tue, 22 Apr 2025 08:40:37 -0700 Subject: [PATCH 6/7] Update test/README.md Co-authored-by: Eemeli Aro --- test/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/README.md b/test/README.md index f046147902..bf3d56b02b 100644 --- a/test/README.md +++ b/test/README.md @@ -7,7 +7,7 @@ These test files are intended to be useful for testing multiple different _messa - `syntax-errors.json` — Strings that should produce a Syntax Error when parsed. > [!NOTE] -> `syntax-errors.json` does not include tests for the disallowed uses of unpaired surrogate code points +> Tests for the disallowed uses of unpaired surrogate code points are not included > because JSON does not permit unpaired surrogate code points. > If your implementation uses UTF-16 based strings (such as JavaScript `String` or Java `java.lang.String`) > or otherwise allows unpaired surrogates in text or literals, you will need to implement tests equivalent From 4d631bad03d57cc879598d8598eae1f35b6ef3fc Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Tue, 22 Apr 2025 08:40:50 -0700 Subject: [PATCH 7/7] Update test/README.md Co-authored-by: Eemeli Aro --- test/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/test/README.md b/test/README.md index bf3d56b02b..b102aa14a9 100644 --- a/test/README.md +++ b/test/README.md @@ -13,7 +13,11 @@ These test files are intended to be useful for testing multiple different _messa > or otherwise allows unpaired surrogates in text or literals, you will need to implement tests equivalent > to the following for syntax errors: > ```json -> { "src": "{\ud800}" } +> { +> "locale": "en-US", +> "src": "{\ud800}", +> "expErrors": [{ "type": "syntax-error" }] +> } > ``` - `data-model-errors.json` - Strings that should produce a Data Model Error when processed.