FEAT-#0000: Implement DataFrame API standard #6928

anmyachev · 2024-02-12T22:45:21Z

What do these changes do?

This is an example of how DataFrame API protocol implementation might look like if it will be added directly to Modin, instead of https://github.com/data-apis/dataframe-api-compat.

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves #?
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

modin/dataframe_api_standard/column_object.py

modin/dataframe_api_standard/tests/namespace/column_from_sequence_test.py

+    kwargs: dict[str, Any],
+) -> None:
+    df = integer_dataframe_1(library)
+    ns = df.__dataframe_namespace__()


modin/dataframe_api_standard/tests/integration/persistedness_test.py

+        pytest.skip("TODO: enable for modin")
+    df1 = integer_dataframe_1(library)
+    df2 = integer_dataframe_2(library)
+    ns = df1.__dataframe_namespace__()


modin/dataframe_api_standard/tests/namespace/to_array_object_test.py

+
+def test_column_to_array_object(library: BaseHandler) -> None:
+    col = integer_dataframe_1(library).col("a")
+    result = np.asarray(col.persist().to_array())


modin/dataframe_api_standard/tests/integration/persistedness_test.py

+    df1 = df.filter(df.col("a") > 1)
+    df2 = df.filter(df.col("a") <= 1)
+
+    df1 = df1.persist()


modin/dataframe_api_standard/dataframe_object.py

+import numpy as np
+from pandas.api.types import is_extension_array_dtype
+
+import modin.dataframe_api_standard


modin/dataframe_api_standard/dataframe_object.py

+        return df.shape  # type: ignore[no-any-return]
+
+    def group_by(self, *keys: str) -> GroupBy:
+        from modin.dataframe_api_standard.group_by_object import GroupBy


modin/dataframe_api_standard/group_by_object.py

+
+import modin.pandas as pd
+from modin.dataframe_api_standard import Namespace
+from modin.dataframe_api_standard.dataframe_object import DataFrame


modin/dataframe_api_standard/group_by_object.py

+        failed_columns = self._df.columns.difference(result.columns)
+        if len(failed_columns) > 0:  # pragma: no cover
+            msg = "Groupby operation could not be performed on columns "
+            +f"{failed_columns}. Please drop them before calling group_by."


Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

dchigarev · 2024-02-13T14:10:28Z

Why we're merging this to our repo instead of https://github.com/data-apis/dataframe-api-compat? Is this already determined that the dataframe-api repo will be abandoned and won't accept contributions?

dchigarev · 2024-02-13T14:30:20Z

In particular, I'm asking about this PR (data-apis/dataframe-api-compat#71), I've seen some fresh comments there from you. Are you expecting this to be merged? If so, do we need the PR in modin?

anmyachev · 2024-02-13T14:33:55Z

@dchigarev Why we're merging this to our repo instead of https://github.com/data-apis/dataframe-api-compat?

I consider this change as temporary in order to be in time for the 0.27 release.

Is this already determined that the dataframe-api repo will be abandoned and won't accept contributions?

I don't think so.

In particular, I'm asking about this PR (data-apis/dataframe-api-compat#71), I've seen some fresh comments there from you. Are you expecting this to be merged? If so, do we need the PR in modin?

I’m still sure that it can be merged, but it definitely won’t make it in time for tomorrow’s release.

dchigarev · 2024-02-13T14:34:37Z

I consider this change as temporary in order to be in time for the 0.27 release.

Why do we want this to be in 0.27 release? cc @YarShev
It looks like nothing pushes us to have a working df-api implementation? If someone is interested to try df-api on Modin, we could provide them @anmyachev's branch with the implementation

anmyachev · 2024-02-13T14:37:40Z

I consider this change as temporary in order to be in time for the 0.27 release.

why do we want this to be in 0.27 release? cc @YarShev

There is an opportunity to provide these changes earlier.

anmyachev · 2024-02-13T14:44:00Z

It is also worth noting that:

The protocol is still in beta and may change significantly. However, almost all the code can be reused from Pandas, given that the public API is the same.
Even if we place the code in dataframe-api-compat repository, this does not mean that it will be fully supported by other developers. Support is still expected from the Modin developers.

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

dchigarev · 2024-02-13T15:16:38Z

Even if we place the code in dataframe-api-compat repository, this does not mean that it will be fully supported by other developers.

Sure, I also thought the same. I'm just worried about how we're going to sync modin's repos with the PR to dataframes-api (data-apis/dataframe-api-compat#71). The diffs are massive, so applying review suggestions for the dataframes-api PR and possibly fixing bugs for the implementation in the Modin at the same time, could result into a severe unsynchrony between these two implementations.

Don't we want to instead spend our efforts on the dataframes-api PR data-apis/dataframe-api-compat#71 (making a review by our team/working on suggestions together) and the future support of modin's implementation there?

anmyachev · 2024-04-17T13:05:17Z

Closed in favour of #7196

anmyachev force-pushed the dataframe-api-impl branch from 72b3b5b to 5ab47be Compare February 12, 2024 22:46

github-advanced-security bot found potential problems Feb 12, 2024

View reviewed changes

anmyachev marked this pull request as ready for review February 12, 2024 22:54

anmyachev requested review from devin-petersohn, mvashishtha, RehanSD, YarShev, vnlitvinov, dchigarev and a team as code owners February 12, 2024 22:55

anmyachev added 6 commits February 13, 2024 15:01

FEAT-modin-project#6890: add consortium standard entrypoint

f6b2f61

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

fixes

2fd81dc

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

add missed pytest call

fdc21a6

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

FEAT-#0000: Implement DataFrame API standard

38630e7

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'dataframe-api-compat'

a11bf8d

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

add license headers

0a7a637

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

anmyachev force-pushed the dataframe-api-impl branch from 5ab47be to 0a7a637 Compare February 13, 2024 14:01

fix

116a737

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

fix docstrings

5f7730b

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

anmyachev closed this Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#0000: Implement DataFrame API standard #6928

FEAT-#0000: Implement DataFrame API standard #6928

anmyachev commented Feb 12, 2024 •

edited

Loading

dchigarev commented Feb 13, 2024

dchigarev commented Feb 13, 2024

anmyachev commented Feb 13, 2024

dchigarev commented Feb 13, 2024 •

edited

Loading

anmyachev commented Feb 13, 2024

anmyachev commented Feb 13, 2024 •

edited

Loading

dchigarev commented Feb 13, 2024 •

edited

Loading

anmyachev commented Apr 17, 2024

FEAT-#0000: Implement DataFrame API standard #6928

FEAT-#0000: Implement DataFrame API standard #6928

Conversation

anmyachev commented Feb 12, 2024 • edited Loading

What do these changes do?

dchigarev commented Feb 13, 2024

dchigarev commented Feb 13, 2024

anmyachev commented Feb 13, 2024

dchigarev commented Feb 13, 2024 • edited Loading

anmyachev commented Feb 13, 2024

anmyachev commented Feb 13, 2024 • edited Loading

dchigarev commented Feb 13, 2024 • edited Loading

anmyachev commented Apr 17, 2024

anmyachev commented Feb 12, 2024 •

edited

Loading

dchigarev commented Feb 13, 2024 •

edited

Loading

anmyachev commented Feb 13, 2024 •

edited

Loading

dchigarev commented Feb 13, 2024 •

edited

Loading