.drop_duplicates() works much slower on Modin than on Pandas

[dchigarev]

Modin's .drop_duplicates implementation works approximately 10x-100x times slower than pandas. We'd like to find some room for improvement here.

import modin.pandas as pd
import pandas
import numpy as np
import timeit

NROWS = 1_000_000
RAND_LOW = 0
RAND_HIGH = 10_000_000

data = {
    "col0": np.tile(
        np.concatenate([np.random.randint(RAND_LOW, RAND_HIGH, NROWS // 100 - 1), [1]]),
        100,
    ),
    "col1": np.tile(
        np.concatenate([np.random.randint(RAND_LOW, RAND_HIGH, NROWS // 100 - 1), [1]]),
        100,
    ),
}

md_df = pd.DataFrame(data)
pd_df = pandas.DataFrame(data)

# drop_duplicate: 
#   input_shape = (1_000_000, 2)
#   output_shape = (10_000, 2)

NRUNS = 5
md_res = timeit.timeit(lambda: md_df.drop_duplicates(), number=NRUNS)
pd_res = timeit.timeit(lambda: pd_df.drop_duplicates(), number=NRUNS)

print(f"Cumulative time of {NRUNS} runs for modin: {md_res}s.") # ~10s.
print(f"Cumulative time of {NRUNS} runs for pandas: {pd_res}s.") # ~0.3s

We may use this issue as an epic to reference all of the perf problems with .drop_duplicates()