Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arrays of decimal128 to join tests #4489

Merged
merged 1 commit into from
Jan 11, 2022

Conversation

kuhushukla
Copy link
Collaborator

Fixes #2134
Signed-off-by: Kuhu Shukla kuhus@nvidia.com

Could also add this test additionally but I think the changes below suffice.

# local sort because of https://github.com/NVIDIA/spark-rapids/issues/84
# After 3.1.0 is the min spark version we can drop this
@ignore_order(local=True)
@pytest.mark.parametrize('data_gen', single_array_gens_sample_with_decimal128, ids=idfn)
@pytest.mark.parametrize('join_type', ['Inner', 'Left', 'Right', 'Cross', 'LeftSemi', 'LeftAnti'], ids=idfn)
def test_broadcast_join_right_arr_dec128(data_gen, join_type):
    def do_join(spark):
        left = two_col_df(spark, data_gen, int_gen, length=500)
        right = two_col_df(spark, data_gen, int_gen, length=100)
        return left.join(broadcast(right), left.b == right.b, join_type)
    assert_gpu_and_cpu_are_equal_collect(do_join, conf=allow_negative_scale_of_decimal_conf)

Signed-off-by: Kuhu Shukla <kuhus@nvidia.com>
@kuhushukla kuhushukla added the task Work required that improves the product but is not user facing label Jan 10, 2022
@kuhushukla kuhushukla added this to the Jan 10 - Jan 28 milestone Jan 10, 2022
@kuhushukla kuhushukla self-assigned this Jan 10, 2022
@razajafri
Copy link
Collaborator

should we add [databricks] to the title?

@kuhushukla
Copy link
Collaborator Author

build

@revans2
Copy link
Collaborator

revans2 commented Jan 11, 2022

@razajafri how insistent are you on databricks? If you are okay without it we can merge it in as is. If you really want it we can change the title and rerun

@kuhushukla
Copy link
Collaborator Author

@razajafri Are we good on this change?

@razajafri
Copy link
Collaborator

@razajafri how insistent are you on databricks? If you are okay without it we can merge it in as is. If you really want it we can change the title and rerun

Sorry, I didn't mean to hold this up just for databricks. we can merge it

@razajafri razajafri merged commit 4807bc1 into NVIDIA:branch-22.02 Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Support Array[Decimal] and struct of Decimal being passed along with join
3 participants