feat: Polars generic dataset (take 3)

[astrojuanlu]

Description

Close gh-110.

Development notes

See past discussion in gh-153 and gh-116.

Not 100 % sure I did the rebase right, will look into it later.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

Lots of test failures that are completely unrelated to this PR. Will have a look later.

[sbrugman]

This dataset would be of great use! Happy to help where possible.

The error that is mostly present is:

TypeError: expected exception must be a BaseException type, not DataSetError

The kedro version compared to main shows changes in kedro.io.core:

kedro-org/kedro@08d3a02...main#diff-236fdc9ebc4cf9a942f1044bec507416ca1ab60918d41177e04b18591fecb40d

This probably has to do with the renaming of DataSetError to DatasetError:

kedro/io/core.py

I'm not well-known with the transition plans for the DataSet names and the parity of kedro and kedro-datasets. Do you think it's possible to move forward?

[astrojuanlu]

Hi @sbrugman, that help would be really appreciated! This has been on my to-do list for too long.

I just rebased and force pushed on top of current main, let's see what the CI says.

[astrojuanlu]

About the move to *Dataset from *DataSet, #255 should take care of that I believe. But it's a bit stuck.

[sbrugman]

LGTM!

[astrojuanlu]

Looks like everything is green now 😄

Have you used it already @sbrugman? To be honest I haven't looked at the code nor tried to use it yet myself. But it looks like

• kedro_datasets/polars/generic_dataset.py has 100 % coverage
• Tests are parametrized by file type and location

I'm... marking this as ready for review 😬

[sbrugman]

As it stands, we will use it for parquet on S3 once this dataset is incorporated in kedro-datasets. Will report issues if we run into them.

[astrojuanlu]

I tested this with https://github.com/astrojuanlu/workshop-jupyter-kedro successfully:

diff --git a/conf/base/catalog.yml b/conf/base/catalog.yml
index 854b5e8..f06dd7e 100644
--- a/conf/base/catalog.yml
+++ b/conf/base/catalog.yml
@@ -12,22 +12,14 @@ openrepair-0_3-categories:
   filepath: data/01_raw/OpenRepairData_v0.3_Product_Categories.csv
 
 openrepair-0_3-combined:
-  type: polars.CSVDataSet
-  filepath: data/02_intermediate/openrepairdata_v0.3_combined.csv
-  load_args:
-    dtypes:
-      product_age: ${pl:Float64}
-      group_identifier: ${pl:Utf8}
-    try_parse_dates: true
+  type: polars.GenericDataSet
+  file_format: parquet
+  filepath: data/02_intermediate/openrepairdata_v0.3_combined.pq
 
 openrepair-0_3:
-  type: polars.CSVDataSet
-  filepath: data/03_primary/openrepairdata_v0.3_clean.csv
-  load_args:
-    dtypes:
-      product_age: ${pl:Float64}
-      group_identifier: ${pl:Utf8}
-    try_parse_dates: true
+  type: polars.GenericDataSet
+  file_format: parquet
+  filepath: data/03_primary/openrepairdata_v0.3_clean.pq
 
 wordcloud-plot:
   type: matplotlib.MatplotlibWriter

and everything worked perfectly. I cannot approve my own pull request 😬 so, reviews welcome!

[merelcht]

I've copied over my comments from the closed Polars PRs which were left unanswered.

[ankatiyar]

I tested it too and it works fine! :)

[SajidAlamQB]

This looks great, just left some minor nitpicks but otherwise all good to go! 🌟

[astrojuanlu]

Polars 0.19 is out 😄 after this PR is merged I'll verify that nothing broke here https://github.com/pola-rs/polars/releases/tag/py-0.19.0

[merelcht]

This looks great! Let's get it merged 👍

[astrojuanlu]

🔥 Thanks a lot reviewers! Super happy to see this land 👏🏽

[astrojuanlu]

And thanks a lot to @wmoreiraa for starting the original effort!

[grofte]

Oh, I thought this was already available in 0.19.1. If I add a pandas<->polars conversion at the beginning and end of every node I could have a polars pipeline, right? And then switch over once a new version is released that reads and writes Polars?

[astrojuanlu]

@grofte This is available in kedro-datasets! You have to pip install kedro-datasets[polars]

[grofte]

I'm running Kedro 0.19.2 and I've done pip install kedro-dataset[polars] (and repeated it after uninstalling kedro-dataset). The kedro-dataset version is 2.1.0.

I kept getting kedro.io.core.DatasetError: Class 'polars.GenericDataSet' not found, is this a typo? but then I found out that S->s so now I get Class 'polars.GenericDataset' not found, is this a typo?. According to this https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-2.0.0/api/kedro_datasets.html GenericDataset is only a thing for Pandas.

EDIT OOOOOOOh. It's EagerPolarsDataset or LazyPolarsDataset to get a generic dataset with Polars.

[astrojuanlu]

Oh, yes it changed its name I believe 🙏🏽 Glad you got it to work! If you have any troubles with it, feel free to open an issue.

[grofte]

Thank you!

And update your blog post 😉 https://kedro.org/blog/a-polars-exploration-into-kedro

It's the premiere source for kedro+polars information 😁

[grofte]

For any future folks that come across this: There's a bug, I think in Polars but maybe in Kedro Datasets, where Kedro's EagerDataset opens the parquet file but polars doesn't recognize it as bytes / io.BufferedIOBase / io.RawIOBase and therefore sends it to .scan_parquet which only takes paths as an argument. I don't know which library it is happening in, or why, so I can't make a PR to fix it but there's a work-around. In your catalog.yml add

load_args:
    use_pyarrow: true

[astrojuanlu]

Ugh, could you open a separate issue about this @grofte ?