Orc read map #3394
Orc read map #3394
Conversation
Signed-off-by: Chong Gao <chongg@nvidia.com>
Signed-off-by: Chong Gao <chongg@nvidia.com>
Signed-off-by: Chong Gao <res_life@163.com>
|
Note: please do not merge, is waiting CUDF to merge: rapidsai/cudf#9132 |
| @@ -82,9 +82,16 @@ def test_basic_read(std_input_path, name, read_func, v1_enabled_list, orc_impl, | |||
| StructGen([['child0', byte_gen], ['child1', orc_basic_struct_gen]]), | |||
| StructGen([['child0', ArrayGen(short_gen)], ['child1', double_gen]])] | |||
|
|
|||
| all_basic_map_gens = [MapGen(f(nullable=False), f()) for f in [BooleanGen, ByteGen, ShortGen, IntegerGen, LongGen, FloatGen, DoubleGen, TimestampGen]] + [simple_string_to_string_map_gen] | |||
| all_basic_map_gens += [DateGen(start=date(1590, 1, 1))] | |||
There was a problem hiding this comment.
seems this is not a Map type?
There was a problem hiding this comment.
The same question. DateGen is obviously not a MapGen. You may refer to the MapGen for Parquet reader tests here https://github.com/NVIDIA/spark-rapids/blob/branch-21.10/integration_tests/src/main/python/parquet_test.py#L45
There was a problem hiding this comment.
fixed, referred parquet map gen
| all_basic_map_gens += [DateGen(start=date(1590, 1, 1))] | ||
| orc_map_gens_sample = all_basic_map_gens + [MapGen(StringGen(pattern='key_[0-9]', nullable=False), ArrayGen(string_gen), max_length=10), | ||
| MapGen(RepeatSeqGen(IntegerGen(nullable=False), 10), long_gen, max_length=10), | ||
| MapGen(StringGen(pattern='key_[0-9]', nullable=False), simple_string_to_string_map_gen)] |
There was a problem hiding this comment.
Does your PR support Map(xx, Struct)? If yes, better add this for supporting.
There was a problem hiding this comment.
added map(struct, struct)
firestarman
added
the
cudf_dependency
|
Move to draft because it depends on the open PR rapidsai/cudf#9132. |
| @@ -82,9 +82,16 @@ def test_basic_read(std_input_path, name, read_func, v1_enabled_list, orc_impl, | |||
| StructGen([['child0', byte_gen], ['child1', orc_basic_struct_gen]]), | |||
| StructGen([['child0', ArrayGen(short_gen)], ['child1', double_gen]])] | |||
|
|
|||
| all_basic_map_gens = [MapGen(f(nullable=False), f()) for f in [BooleanGen, ByteGen, ShortGen, IntegerGen, LongGen, FloatGen, DoubleGen, TimestampGen]] + [simple_string_to_string_map_gen] | |||
| all_basic_map_gens += [DateGen(start=date(1590, 1, 1))] | |||
There was a problem hiding this comment.
The same question. DateGen is obviously not a MapGen. You may refer to the MapGen for Parquet reader tests here https://github.com/NVIDIA/spark-rapids/blob/branch-21.10/integration_tests/src/main/python/parquet_test.py#L45
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Chong Gao <res_life@163.com>
Signed-off-by: Chong Gao <res_life@163.com>
|
This is likely to conflict with #3393, so this PR will need to be updated if 3393 is merged before this one. |
|
build |
|
LGTM |
|
build |
orc read map
This fixes #3297
Signed-off-by: Chong Gao res_life@163.com