Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set the default charset as UTF-8 for REST calls with application/json MediaType #1437

Closed
lqiu96 opened this issue Mar 3, 2023 · 0 comments · Fixed by #1477 or #1571
Closed

Set the default charset as UTF-8 for REST calls with application/json MediaType #1437

lqiu96 opened this issue Mar 3, 2023 · 0 comments · Fixed by #1477 or #1571
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@lqiu96
Copy link
Contributor

lqiu96 commented Mar 3, 2023

Discovered while implementing the Showcase Compliance Suite:

2023/03/03 17:19:42   request: {
  "name":  "Extreme values",
  "info":  {
    "fString":  "non-ASCII+non-printable string ? ? ? \"\\/\b\f\r\t? works, not newlines yet",
    "fInt32":  2147483647,
    "fSint32":  2147483647,
    "fSfixed32":  2147483647,
    "fUint32":  4294967295,
    "fFixed32":  4294967295,
    "fInt64":  "9223372036854775807",
    "fSint64":  "9223372036854775807",
    "fSfixed64":  "9223372036854775807",
    "fUint64":  "18446744073709551615",
    "fFixed64":  "18446744073709551615",
    "fDouble":  1.7976931348623157e+308,
    "fFloat":  3.4028235e+38,
    "fBool":  false,
    "fBytes":  "",
    "fKingdom":  "LIFE_KINGDOM_UNSPECIFIED",
    "fChild":  null,
    "pString":  "Goodbye",
    "pInt32":  2147483647,
    "pDouble":  1.7976931348623157e+308,
    "pBool":  false
  },
  "serverVerify":  true,
  "fInt32":  0,
  "fInt64":  "0",
  "fDouble":  0
}
Received Info:  f_string:"non-ASCII+non-printable string ? ? ? \"\\/\x08\x0c\r\t? works, not newlines yet"  f_int32:2147483647  f_sint32:2147483647  f_sfixed32:2147483647  f_uint32:4294967295  f_fixed32:4294967295  f_int64:9223372036854775807  f_sint64:9223372036854775807  f_sfixed64:9223372036854775807  f_uint64:18446744073709551615  f_fixed64:18446744073709551615  f_double:1.7976931348623157e+308  f_float:3.4028235e+38  p_string:"Goodbye"  p_int32:2147483647  p_double:1.7976931348623157e+308  p_bool:false
Expected Info:  f_string:"non-ASCII+non-printable string ☺ → ← \"\\/\x08\x0c\r\tሴ works, not newlines yet"  f_int32:2147483647  f_sint32:2147483647  f_sfixed32:2147483647  f_uint32:4294967295  f_fixed32:4294967295  f_int64:9223372036854775807  f_sint64:9223372036854775807  f_sfixed64:9223372036854775807  f_uint64:18446744073709551615  f_fixed64:18446744073709551615  f_double:1.7976931348623157e+308  f_float:3.4028235e+38  p_string:"Goodbye"  p_int32:2147483647  p_double:1.7976931348623157e+308  p_bool:false
2023/03/03 17:19:42 server error: (ComplianceSuiteRequestMismatchError) contents of request "Extreme values" do not match test suite

The unicode values aren't send properly as part of the request. The correct values should be ☺ → ← but they get sent as ? ? ?.

Issue:

  1. The MediaType for REST requests is set here:
    https://github.com/googleapis/gapic-generator-java/blob/c23f981e2ac3c573bed51e725dc7061551179400/gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/HttpRequestRunnable.java#L175

It only set the MediaType's charset as application/json when it could be application/json; charset=utf-8. This ensures that the unicode values are set to be encoded properly.

  1. The default charset is not set to be UTF-8:
    https://github.com/googleapis/google-http-java-client/blob/84216c5cfe2f1600464d34661a208cf165e11b9b/google-http-client/src/main/java/com/google/api/client/http/AbstractHttpContent.java#L94-L98

Instead, it's set to StandardCharsets.ISO_8859_1, which only represents the first 256 characters of Unicode.

This seems to due to this comment: googleapis/google-http-java-client#300 (comment)

@lqiu96 lqiu96 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Mar 3, 2023
@lqiu96 lqiu96 self-assigned this Mar 3, 2023
@lqiu96 lqiu96 transferred this issue from googleapis/google-http-java-client Mar 6, 2023
@lqiu96 lqiu96 changed the title Use Charset UTF-8 for application/json MedaType Set the default charset as UTF-8 for REST calls with application/json MediaType Mar 9, 2023
gcf-merge-on-green bot pushed a commit that referenced this issue Mar 20, 2023
Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://github.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes #1437
lqiu96 added a commit to renovate-bot/gapic-generator-java that referenced this issue Mar 20, 2023
)

Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://github.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes googleapis#1437
lqiu96 added a commit to renovate-bot/gapic-generator-java that referenced this issue Mar 21, 2023
)

Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://github.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes googleapis#1437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
1 participant