Fix CAST(JSON as ROW(ARRAY)) #9447
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
✅ Deploy Preview for meta-velox canceled.
|
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
3244566
to
08a94a3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
08a94a3
to
829d20c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Differential Revision: D56013293
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Differential Revision: D56013293
Yuhta
approved these changes
Apr 11, 2024
| // Mapping from lower-case field names of the target RowType to their | ||
| // indices. | ||
| folly::F14FastMap<std::string, column_index_t> fieldIndices; | ||
| for (auto i = 0; i < rowType.size(); ++i) { |
There was a problem hiding this comment.
nit: RowType::size is virtual and may not be able to inline here, so better assign it to a variable before loop
| fieldIndices[key] = i; | ||
| } | ||
|
|
||
| std::vector<bool> foundIndices(rowType.size(), false); |
There was a problem hiding this comment.
We could initialize fieldIndices value to -1 then we don't need this second memory.
There was a problem hiding this comment.
Sounds good. Updated.
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: Yuhta Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
829d20c
to
afac6c4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
|
@Yuhta Jimmy, thank you for review. Updated to address your comments. |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: Yuhta Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
afac6c4
to
e84d832
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: Yuhta Differential Revision: D56013293
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: Yuhta Differential Revision: D56013293
Yuhta
approved these changes
Apr 11, 2024
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
|
Please rebase to main after #9451 lands. |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
e84d832
to
77fa564
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
mbasmanova
force-pushed
the
export-D56013293
branch
from
77fa564
to
efeb0de
Compare
mbasmanova
added a commit
to mbasmanova/velox-1
that referenced
this pull request
Apr 11, 2024
Summary: CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293
|
This pull request was exported from Phabricator. Differential Revision: D56013293 |
|
This pull request has been merged in 707dbfd. |
yanngyoung
pushed a commit
to yanngyoung/velox
that referenced
this pull request
Apr 12, 2024
Summary: Pull Request resolved: facebookincubator#9447 CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293 fbshipit-source-id: 28337280e1943bca4d6b46ff6f1f62341656bdda
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
Jun 7, 2024
Summary: Pull Request resolved: facebookincubator#9447 CAST(JSON as ROW(ARRAY()) used to fail with ``` OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered. ``` According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding. Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates. Also, fix the test to actually verify JSON objects with mixed case keys. Reviewed By: xiaoxmeng, Yuhta Differential Revision: D56013293 fbshipit-source-id: 28337280e1943bca4d6b46ff6f1f62341656bdda
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
CAST(JSON as ROW(ARRAY()) used to fail with
According to simdjson documentation, https://github.com/simdjson/simdjson/blob/master/doc/basics.md, it is not allowed to store object values for later processing. These must be consumed or copied before proceeding.
Also, fixed behavior when JSON object contains duplicate keys. Presto throws, but previous implementation used to allow duplicates.
Also, fix the test to actually verify JSON objects with mixed case keys.
Differential Revision: D56013293