Strict mode for JSON parsing, contributed by @marten-voorberg. #2437
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Feat #6: Add strict flag to Gson and GsonBuilder * Test #2: Add failing tests for capitalized keywords * Feat #2: JsonReader does not read (partially) capitalized keywords if strict mode is used * Feat #3: Added implementation and tests for JSONReader not accepting specific escape sequence representing in strict mode * Test #3: Simplify test cases by removing unnecessary array * Feat #3: Improve error by including the illegal character * Feat #5: JsonReader does not allow unespaced control flow characters in strict mode * Test #5: Test unespaced control flow characters in strict mode * Feat #4: Disallow espaced newline character in strict mode * Test #4: Add tests for (dis)allowing newline character depensding on strictness * Test #5: Test case for unescaped control char in non-strict mode * Test #2: Simplify test cases * Feat #13: Change leniency API to Strictness enum in JsonReader, Gson, and GsonBuilder * Feat #15: Change JsonWriter API to also use Strictness * Test #15: Test Strictness in JsonWriter API * Doc #15: Add and update documentation for Strictness in JsonWriter API * refactor #12: Fixed typos and empty catch brackets in tests * refactor #12: Resolved importing wildcards, made some lines adhere to Google java style * #5 Add test case for unescaped control characters * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <2295699210@qq.com> Co-authored-by: Marten Voorberg <martenvoorberg@gmail.com> * Doc #17: Add and change javadoc of public methods * Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder * Test #34: Add tests for setting strictness through GsonBuilder * Fix: Add Fix broken test * Fix: Invalid JavaDoc in Gson.java * Doc #17: update outdated javadoc * #37: Resolve more PR feedback * Fix #37: Resolve various PR comments * Fix #37: Resolve various PR comments * Refactor #35: Refactor JsonReader#peekKeyword to reduce the amount of strictness checks (#39) * Doc #40: Update JavaDoc based on PR feedback * Doc #40: Update old RFC in GsonBuilder documentation * Doc #40: Fix formatting error in JavaDoc * Doc #40: Add tests for setting strictness and lenient to JsonReaderTest * Test #43: Changed tests to make use of assertThrows * test #43: Changed tests to make use of assertThrows as per feedback * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * Test #43: Update JsonWriterTest#testStrictnessNull to use assertThrows * test #43: Resolve PR recommendations * Test #43: Mini change to TC * Test #43: Mini change to TC --------- Co-authored-by: Marten Voorberg <martenvoorberg@gmail.com> * doc #46: Resolved comments in main PR * Feat #45: Change Gson.fromJson and Gson.toJson to be strict when the provided writer/reader is strict * Fix #45: Small type * Update gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Fix #45: Resolve various comments by Marcono1234 * Update gson/src/main/java/com/google/gson/GsonBuilder.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Fix #45: Resolve various comments by Marcono1234 * Fix #45: Resolve various comments by eamonmcmanus * Strictness mode follow-up * Update Troubleshooting.md and Gson default lenient mode documentation * Always use GSON strictness when set. * Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT * Update JavaDoc with new strictness functionality * Replace default with legacy strict for JsonReader javadoc * Add JSONReader test cases for U2028 and U2029 * Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion * Deprecate setLenient in favor of setStrictness --------- Co-authored-by: Carl Peterson <unknown> Co-authored-by: Gustaf Johansson <gustajoh@kth.se> Co-authored-by: gustajoh <58432871+gustajoh@users.noreply.github.com> Co-authored-by: LMC117 <2295699210@qq.com> Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com>
Temporarily remove some new deprecations since they cause compilation failures because of `-Werror`. Adjust some exception message assertions.
* Strictness mode follow-up - Remove mentions of `null` Gson strictness; this is an implementation detail - Fix incorrect / outdated documentation - Reduce links to RFC; if there is already a link to it in a previous sentence don't link to it again - Extend and update tests - Minor punctuation changes in documentation for consistency * Deprecate `setLenient` methods
Marcono1234
reviewed
Jul 18, 2023
There was a problem hiding this comment.
Found a few more minor things (most likely I was also responsible for them in the first place).
Out of curiosity, can you see the details why the Google CLA bot fails for one of the Co-authored-by fields?
I could imagine that one of these is causing the issue:
Co-authored-by: Carl Peterson <unknown>Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Not sure if[and]are permitted in e-mail addresses, respectively whether the Google CLA bot supports them.
In case it complains about Carl Peterson <unknown>, interestingly it was apparently happily accepting that as author, see https://github.com/google/gson/runs/11620478807:
(Though maybe the Google CLA bot implementation was changed in the meantime, or this is a bug?)
Can we ignore this CLA bot issue? For context, it appears the commit with the missing e-mail address originates from marten-voorberg#9.
Marcono1234
reviewed
Jul 19, 2023
|
@eamonnmcmanus for simplicity I have addressed the review comments above in #2456, which targets this |
* Adjust ProGuard default rules and shrinking tests (#2420) * Adjust ProGuard default rules and shrinking tests * Adjust comment * Add shrinking test for class without no-args constructor; improve docs * Improve Unsafe mention in Troubleshooting Guide * Improve comment for `-if class *` * Bump com.google.guava:guava from 32.0.1-jre to 32.1.1-jre (#2444) Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.google.guava:guava-testlib from 32.0.1-jre to 32.1.1-jre (#2443) Bumps [com.google.guava:guava-testlib](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava-testlib dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support non-generic type for `TypeToken.getParameterized` for legacy reasons (#2447) This partially restores the behavior before a589ef2, except that back then for a non-generic type a bogus `TypeToken(ParameterizedType)` was created, whereas now a `TypeToken(Class)` is created instead. * Fixed Typo in GsonBuilder.java (#2449) * Make date-formatting tests less fragile with regular expressions. (#2450) * Make date-formatting tests less fragile with regular expressions. This is not great. We should really ensure that formatted dates are the same regardless of JDK version. There is code that attempts to do that but it is not really effective. So for now we fudge around the differences by using regular expressions to paper over the differences. * Temporarily add test-debugging code. * Another attempt at debugging a test failure. * Fix pattern in assertion. * Modification in test cases (#2454) * Fixed Typo in GsonBuilder.java * Suggestions on Test cases * Modified test cases using assertThrows method (JUnit) * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/GsonTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> --------- Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Minor follow-up changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <97422844+elevne@users.noreply.github.com> Co-authored-by: Éamonn McManus <emcmanus@google.com> Co-authored-by: Wonil <cwi5525@naver.com>
|
I think this is ready to go. Many thanks to @marten-voorberg and @Marcono1234! The CLA complaint was indeed due to the commit from Carl Peterson. At this point it is two small test methods in |
This was referenced Aug 4, 2023
tibor-universe
pushed a commit
to getuniverse/gson
that referenced
this pull request
Sep 14, 2024
…e#2437) * Strict mode for JSON parsing (google#2323) * Feat #6: Add strict flag to Gson and GsonBuilder * Test #2: Add failing tests for capitalized keywords * Feat #2: JsonReader does not read (partially) capitalized keywords if strict mode is used * Feat #3: Added implementation and tests for JSONReader not accepting specific escape sequence representing in strict mode * Test #3: Simplify test cases by removing unnecessary array * Feat #3: Improve error by including the illegal character * Feat #5: JsonReader does not allow unespaced control flow characters in strict mode * Test #5: Test unespaced control flow characters in strict mode * Feat #4: Disallow espaced newline character in strict mode * Test #4: Add tests for (dis)allowing newline character depensding on strictness * Test #5: Test case for unescaped control char in non-strict mode * Test #2: Simplify test cases * Feat #13: Change leniency API to Strictness enum in JsonReader, Gson, and GsonBuilder * Feat #15: Change JsonWriter API to also use Strictness * Test #15: Test Strictness in JsonWriter API * Doc #15: Add and update documentation for Strictness in JsonWriter API * refactor #12: Fixed typos and empty catch brackets in tests * refactor #12: Resolved importing wildcards, made some lines adhere to Google java style * #5 Add test case for unescaped control characters * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Feat #5: add new lines to make JsonReader able to detect unescaped control characters (U+0000 through U+001F) and throw exceptions. * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue * Test #11: Added two tests for testing implementation of control character handling in strict mode and moved the implementation to nextQuotedValue --------- Co-authored-by: LMC117 <2295699210@qq.com> Co-authored-by: Marten Voorberg <martenvoorberg@gmail.com> * Doc #17: Add and change javadoc of public methods * Doc #17: Update JavaDoc in JsonReader and Strictness * Doc #17: Update JavaDoc in Gson and GsonBuilder * Test google#34: Add tests for setting strictness through GsonBuilder * Fix: Add Fix broken test * Fix: Invalid JavaDoc in Gson.java * Doc #17: update outdated javadoc * google#37: Resolve more PR feedback * Fix google#37: Resolve various PR comments * Fix google#37: Resolve various PR comments * Refactor google#35: Refactor JsonReader#peekKeyword to reduce the amount of strictness checks (google#39) * Doc google#40: Update JavaDoc based on PR feedback * Doc google#40: Update old RFC in GsonBuilder documentation * Doc google#40: Fix formatting error in JavaDoc * Doc google#40: Add tests for setting strictness and lenient to JsonReaderTest * Test google#43: Changed tests to make use of assertThrows * test google#43: Changed tests to make use of assertThrows as per feedback * Test google#43: Update JsonWriterTest#testStrictnessNull to use assertThrows * Test google#43: Update JsonWriterTest#testStrictnessNull to use assertThrows * test google#43: Resolve PR recommendations * Test google#43: Mini change to TC * Test google#43: Mini change to TC --------- Co-authored-by: Marten Voorberg <martenvoorberg@gmail.com> * doc google#46: Resolved comments in main PR * Feat google#45: Change Gson.fromJson and Gson.toJson to be strict when the provided writer/reader is strict * Fix google#45: Small type * Update gson/src/test/java/com/google/gson/stream/JsonReaderTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Fix google#45: Resolve various comments by Marcono1234 * Update gson/src/main/java/com/google/gson/GsonBuilder.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Fix google#45: Resolve various comments by Marcono1234 * Fix google#45: Resolve various comments by eamonmcmanus * Strictness mode follow-up * Update Troubleshooting.md and Gson default lenient mode documentation * Always use GSON strictness when set. * Rename Strictness.DEFAULT to Strictness.LEGACY_STRICT * Update JavaDoc with new strictness functionality * Replace default with legacy strict for JsonReader javadoc * Add JSONReader test cases for U2028 and U2029 * Refactor JSONReader#peekKeyWord() based on @eamonmcmanus's suggestion * Deprecate setLenient in favor of setStrictness --------- Co-authored-by: Carl Peterson <unknown> Co-authored-by: Gustaf Johansson <gustajoh@kth.se> Co-authored-by: gustajoh <58432871+gustajoh@users.noreply.github.com> Co-authored-by: LMC117 <2295699210@qq.com> Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Strictness follow-up (google#2408) * Strictness mode follow-up - Remove mentions of `null` Gson strictness; this is an implementation detail - Fix incorrect / outdated documentation - Reduce links to RFC; if there is already a link to it in a previous sentence don't link to it again - Extend and update tests - Minor punctuation changes in documentation for consistency * Deprecate `setLenient` methods * `strictness2` fixes & improvements (google#2456) * Adjust ProGuard default rules and shrinking tests (google#2420) * Adjust ProGuard default rules and shrinking tests * Adjust comment * Add shrinking test for class without no-args constructor; improve docs * Improve Unsafe mention in Troubleshooting Guide * Improve comment for `-if class *` * Bump com.google.guava:guava from 32.0.1-jre to 32.1.1-jre (google#2444) Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.google.guava:guava-testlib from 32.0.1-jre to 32.1.1-jre (google#2443) Bumps [com.google.guava:guava-testlib](https://github.com/google/guava) from 32.0.1-jre to 32.1.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava-testlib dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support non-generic type for `TypeToken.getParameterized` for legacy reasons (google#2447) This partially restores the behavior before a589ef2, except that back then for a non-generic type a bogus `TypeToken(ParameterizedType)` was created, whereas now a `TypeToken(Class)` is created instead. * Fixed Typo in GsonBuilder.java (google#2449) * Make date-formatting tests less fragile with regular expressions. (google#2450) * Make date-formatting tests less fragile with regular expressions. This is not great. We should really ensure that formatted dates are the same regardless of JDK version. There is code that attempts to do that but it is not really effective. So for now we fudge around the differences by using regular expressions to paper over the differences. * Temporarily add test-debugging code. * Another attempt at debugging a test failure. * Fix pattern in assertion. * Modification in test cases (google#2454) * Fixed Typo in GsonBuilder.java * Suggestions on Test cases * Modified test cases using assertThrows method (JUnit) * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/GsonTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonArrayAsListTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/JsonStreamParserTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/TypeAdapterTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Update gson/src/test/java/com/google/gson/ToNumberPolicyTest.java Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> --------- Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> * Minor follow-up changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <97422844+elevne@users.noreply.github.com> Co-authored-by: Éamonn McManus <emcmanus@google.com> Co-authored-by: Wonil <cwi5525@naver.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Marten <martenvoorberg@gmail.com> Co-authored-by: Gustaf Johansson <gustajoh@kth.se> Co-authored-by: gustajoh <58432871+gustajoh@users.noreply.github.com> Co-authored-by: LMC117 <2295699210@qq.com> Co-authored-by: Marcono1234 <Marcono1234@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: elevne <97422844+elevne@users.noreply.github.com> Co-authored-by: Wonil <cwi5525@naver.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.