-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: extensible function registry #2699
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems reasonable.
- this needs automated tests.
- should our other functions migrate over as well?
There's really nothing to test so far that's not already covered by all of tests. This is all internal. If things were to have broke, our existing tests would have caught known any regressions.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we add an SLT for the cosine similarity function just to make sure it's all wired up correctly (even if it needs to be skipped for rpc/flight).
@scsmithr we already have a bunch of tests for it. |
Let's hold off on merging this until we decide if #2715 is worth cutting a quick 0.9.1 over. |
ps: this is cool! |
I didn't want to create 1 massive undecipherable PR, so I wanted to submit this first.
This ties a
FunctionRegistry
instance to a session instead of the previously globalFUNCTION_REGISTRY
.Right now, the entries are written to both the catalog and the function registry.
I'd prefer writing to just one or the other, but havent quite figured that out yet as our udfs are not serializable so they cant be written directly to the catalog.
Some other the notable changes so far
register_function
on the session.cosine_similarity
to the new datafusion udf impl pattern for easier usage with the newregister_function
method.I manually tested this by removing cosine_similarity from the builtin functions and registering it manually
This works as expected in local sessions only. We'll have to modify the planner to handle udfs in a hybrid context at a later date.
Next steps.
I'm going to try to integrate the work I did with this datafusion <--> polars POC in a subsequent PR.
https://github.com/universalmind303/polar-fusion