-
Notifications
You must be signed in to change notification settings - Fork 901
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft PR for issue 1946 - Snowpark dataset #2032
Conversation
Updated snowpark test prerequisites Draft implementation of ShowParkDataSet class
Also as the PR description at the top says, we'll actually have to move all of this to the new kedro-dataset standalone package. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think in general we shouldn't be doing Pandas stuff in here, it should be a Snowpark dataset first and foremost the same was as we do with Spark - the user can transcode to Pandas later if they need to.
|
||
missing_module = res[0] | ||
|
||
if KNOWN_PIP_INSTALL.get(missing_module): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the existing way to do this is to edit setup.py
and provide optional dependencies.
return DataSetError(f"{DRIVER_ERROR_MESSAGE}{missing_module_instruction}") | ||
|
||
|
||
class SnowflakeTableDataSet(AbstractDataSet[pd.DataFrame, pd.DataFrame]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
class SnowflakeTableDataSet(AbstractDataSet[pd.DataFrame, pd.DataFrame]): | |
class SnowparkTableDataSet(AbstractDataSet[pd.DataFrame, pd.DataFrame]): |
Hi, I am a Field CTO for Datascience at Snowflake and really like the idea of having an integration with Snowpark. You did an amazing job so far and I am happy to support where I can. |
@sfc-gh-mgorkow delighted to see you here! Keen to make some noise about this once it launches :) |
Development of snowpark integration moved to this PR: kedro-org/kedro-plugins#78 |
Description
Addresing comments from previous draft PR
Development notes
Draft implementation of SnowParkDataSet kedro/extras/datasets/snowflake/snowpark_dataset.py
Checklist
RELEASE.md
file