[HUDI-5829] Optimize conversion from json to row format when sanitizing field names #11941
+1,451
−323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vamsikarnika
force-pushed
the
json_to_row
branch
from
c86f236
to
1538426
Compare
vamsikarnika
changed the title
vamsikarnika
force-pushed
the
json_to_row
branch
from
e941544
to
a82fa20
Compare
added 2 commits
September 16, 2024 13:33
vamsikarnika
commented
Sep 16, 2024
Comment on lines
-313
to
-325
| // As we don't do rounding, the validation will enforce the scale part and the integer part are all within the | ||
| // limit. As a result, if scale is 2 precision is 5, we only allow 3 digits for the integer. | ||
| // Allowed: 123.45, 123, 0.12 | ||
| // Disallowed: 1234 (4 digit integer while the scale has already reserved 2 digit out of the 5 digit precision) | ||
| // 123456, 0.12345 | ||
| if (bigDecimal.scale() > decimalType.getScale() | ||
| || (bigDecimal.precision() - bigDecimal.scale()) > (decimalType.getPrecision() - decimalType.getScale())) { | ||
| // Correspond to case | ||
| // org.apache.avro.AvroTypeException: Cannot encode decimal with scale 5 as scale 2 without rounding. | ||
| // org.apache.avro.AvroTypeException: Cannot encode decimal with scale 3 as scale 2 without rounding | ||
| return Pair.of(false, null); | ||
| } | ||
|
|
There was a problem hiding this comment.
Have moved this code to DecimalFieldProcessor class, so this can be reused by both JsonToAvro and JsonToRow processors.
vamsikarnika
commented
Sep 16, 2024
Comment on lines
+41
to
+46
| static Stream<Object> decimalBadCases() { | ||
| return Stream.of( | ||
| // Invalid schema definition. | ||
| Arguments.of(DECIMAL_AVRO_FILE_INVALID_PATH, "123.45", null, false), | ||
| // Schema set precision as 5, input overwhelmed the precision. | ||
| Arguments.of(DECIMAL_AVRO_FILE_PATH, "123333.45", null, false), |
There was a problem hiding this comment.
Have moved the test data generators from TestMercifulJsonConvertor to this base class, so that both row and avro conversions can use same data for testing.
vamsikarnika
commented
Sep 16, 2024
Comment on lines
-76
to
+71
| @ValueSource(strings = { | ||
| "{\"first\":\"John\",\"last\":\"Smith\"}", | ||
| "[{\"first\":\"John\",\"last\":\"Smith\"}]", | ||
| "{\"first\":\"John\",\"last\":\"Smith\",\"suffix\":3}", | ||
| }) | ||
| @MethodSource("dataNestedJsonAsString") |
There was a problem hiding this comment.
Have converted this MethodSource and moved it Test Base class to be reused by MercifulJsonToRowConverter
vamsikarnika
force-pushed
the
json_to_row
branch
from
2c511f1
to
857f45b
Compare
jonvex
requested changes
Sep 30, 2024
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Outdated
Show resolved
Hide resolved
|
|
||
| import org.apache.hudi.exception.HoodieJsonConversionException; | ||
|
|
||
| public class HoodieJsonToRowConversionException extends HoodieJsonConversionException { |
There was a problem hiding this comment.
Created https://issues.apache.org/jira/browse/HUDI-8279 as follow-up
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Show resolved
Hide resolved
...ties/src/main/java/org/apache/hudi/utilities/sources/helpers/MercifulJsonToRowConverter.java
Outdated
Show resolved
Hide resolved
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Show resolved
Hide resolved
| } | ||
| } | ||
|
|
||
| public Either<Row, String> fromJsonToRowWithError(String json) { |
There was a problem hiding this comment.
Explain method return in comment. I think the name could be more clear. fromJsonToRowForErrortable? fromJsonToRowSafe?
There was a problem hiding this comment.
Added the comment. I've followed similar naming as AvroConverter, since it returns a pair of Row and Error string, I think fromJsonToRowWithError makes sense.
added 2 commits
October 1, 2024 17:36
vamsikarnika
force-pushed
the
json_to_row
branch
from
98f6dde
to
ef80959
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Currently when source data has to read in row format and sanitization is enabled, we first read the data in avro format(which supports sanitization) and later convert from avro to row. This new approach simplifies this process by directly converting from json to row while applying sanitization.
Impact
When source data has to be read in row format, and sanitization is enabled. This change should make the conversion from json to row faster by directly converting from json to row.
This change directly affects the existing streams. This is currently added behind a flag which can be disabled if any issues are to be found.
Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist