Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT-#7203: Make sure modin works correctly with pandas, which uses pyarrow as a backend #7204

Merged
merged 59 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8e46e4e
TEST-#7049: Add some sanity tests with pyarrow-backed pandas dataframes
anmyachev Mar 11, 2024
6814c6e
fixes
anmyachev Apr 2, 2024
e1dbc69
fix
anmyachev Apr 19, 2024
0241d7f
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev Apr 19, 2024
7b925a5
cleanup
anmyachev Apr 19, 2024
23003c5
fix comment
anmyachev Apr 19, 2024
cc2a5ab
skip some cases for HDK
anmyachev Apr 19, 2024
b710865
FEAT-#7203: Make sure modin works correctly with pandas, which uses p…
anmyachev Apr 19, 2024
310f12a
don't use numpy types directly
anmyachev Apr 19, 2024
cb90479
try another dtype_backend
anmyachev Apr 19, 2024
f9b2560
fix
anmyachev Apr 19, 2024
907fc9a
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev Apr 29, 2024
13e0d0e
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 2, 2024
ddcda4f
fixes
anmyachev May 2, 2024
afae62f
fix
anmyachev May 2, 2024
316cddb
fix
anmyachev May 2, 2024
639c2ed
fix pivot_table
anmyachev May 2, 2024
05f32e5
fix
anmyachev May 2, 2024
194cc68
find potential problem areas at the query_compiler level
anmyachev May 3, 2024
91f2607
some more places
anmyachev May 3, 2024
e24201f
add construct_dtype
anmyachev May 3, 2024
4dba613
fix
anmyachev May 6, 2024
5f56c4a
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 6, 2024
ea05389
fix
anmyachev May 6, 2024
005f480
fix
anmyachev May 6, 2024
0d34bea
fix
anmyachev May 6, 2024
aac7097
fix
anmyachev May 6, 2024
258c3b9
fix
anmyachev May 6, 2024
068f67d
fix
anmyachev May 6, 2024
45c1d1f
fix
anmyachev May 7, 2024
b114314
fix
anmyachev May 7, 2024
c597f7f
cleanup
anmyachev May 7, 2024
9562144
updates
anmyachev May 10, 2024
46df4ea
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 12, 2024
8b93500
fixes after merge
anmyachev May 12, 2024
ae861e3
new approach
anmyachev May 12, 2024
5b18cfd
cleanup
anmyachev May 12, 2024
9c6ce78
cleanup
anmyachev May 12, 2024
a04b0a2
cleanup
anmyachev May 13, 2024
60101b5
fix
anmyachev May 13, 2024
07f9927
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 13, 2024
c25a419
cleanup
anmyachev May 13, 2024
6e0c37e
fixes
anmyachev May 13, 2024
778be02
cleanup
anmyachev May 13, 2024
9d6d839
cleanup
anmyachev May 13, 2024
22f2db6
cleanup
anmyachev May 13, 2024
acc20b3
cleanup
anmyachev May 13, 2024
7a91fc4
cleanup
anmyachev May 13, 2024
d31e93f
revert changes in metadata/dtypes.py
anmyachev May 13, 2024
a4c5f91
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 13, 2024
b3179fc
fix tests
anmyachev May 13, 2024
18eec16
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 14, 2024
b3471ff
cleanup
anmyachev May 14, 2024
e538fb4
Merge branch 'main' of https://github.com/modin-project/modin into is…
anmyachev May 14, 2024
14b4dd3
fix
anmyachev May 14, 2024
7abfc42
Apply suggestions from code review
anmyachev May 14, 2024
30d4749
Apply suggestions from code review
anmyachev May 14, 2024
3213194
address review comments
anmyachev May 14, 2024
45acef9
expand comments
anmyachev May 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions modin/core/dataframe/algebra/binary.py
Original file line number Diff line number Diff line change
Expand Up @@ -269,8 +269,6 @@ def try_compute_new_dtypes(

try:
if infer_dtypes == "bool" or is_bool_dtype(result_dtype):
# FIXME: https://github.com/modin-project/modin/issues/7203
# can be `pandas.api.types.pandas_dtype("bool[pyarrow]")` depending on the data
dtypes = maybe_build_dtypes_series(
first, second, dtype=pandas.api.types.pandas_dtype(bool)
)
Expand Down
4 changes: 2 additions & 2 deletions modin/core/dataframe/base/dataframe/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@
from enum import Enum
from typing import Dict, List, Sequence, Tuple, cast

import numpy as np
import pandas
from pandas._typing import IndexLabel
from pandas.api.types import is_scalar
from pandas.core.dtypes.common import is_integer_dtype


class Axis(Enum): # noqa: PR01
Expand Down Expand Up @@ -170,7 +170,7 @@ def is_trivial_index(index: pandas.Index) -> bool:
return True
if isinstance(index, pandas.RangeIndex):
return index.start == 0 and index.step == 1
if not (isinstance(index, pandas.Index) and index.dtype == np.int64):
if not (isinstance(index, pandas.Index) and is_integer_dtype(index)):
return False
return (
index.is_monotonic_increasing
Expand Down
Loading
Loading