-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simple Functions #12635
Comments
FYI: I am doing some experiments how this could look like |
I think the idea of making it easier to write functions that include specialized implementations for different types is a great idea. This would likely both make our code faster (more specializations, more uniformly handle constants) as well as reduce the replication (there are several different patterns for dispatch / specialization now -- consolidating would be really nice) |
here is some prior work on the precompiled regular expressions: #11146 (note compiling the regex hasn't ever shown up in any performance trace I did, probably because the cost of actually matching is so much bigger than the cost of compiling and the cost of compiling is spread over 8192 rows |
Thanks @findepi I think this process go through iterations, and easier than was before but still far from perfect. The ScalarUDFImpl common trait is already a huge help, but for example dynamic typing, like Rust can benefit from macros and generate functions with all possible output types combinations, but this is just 1 problem. I'm totally down if we can make it easier |
Just to be clear, what I am imagining comes out of this work is:
|
FYI i touched upon the topic of types on DataFusion meetup in Belgrade yesterday. |
Is your feature request related to a problem or challenge?
Verbosity
Currently implementing a scalar function is a pretty involved process. For example a simple function calculating greatest common divisor looks like this (a lot of detail omitted for brevity):
This function is still "relatively simple" because:
return_type
and again ininvoke
x
or constanty
parametersmake_scalar_function
helper expands scalar constant argument into an array, thusgcd(x, 5)
is less efficient thangcd(x, y)
because it first needs to allocate a temporary buffer for5
s repeated times the batch lengthgcd
on partitioning keys, but this is just an exampleIt should be possible to express functions using simple row-level operations, because in query engines most function logic is structured like that anyway (
compute_gcd
in the example).Local data structures
Currently DataFusion functions are singletons plugged into the execution engine. They have no way to store and reuse buffers or compiled regular expressions, etc.
Thread local storage is not an option because DataFusion, when embedded, does not control creation of application threads.
Describe the solution you'd like
Simple
It should be possible to separate function logic from the mechanics.
Exemplary GCD function needs to provide
fn compute_gcd(x: i64, y: i64) -> Result<i64>
and the rest of boilerplate should not be hand-written for every function separately.It should be possible to implement a function that accepts string values, without having to deal with the 5 different runtime representations that can carry string values:
StringArray, LargeStringArray, StringViewArray, DictionaryArray<Key>, RunArray<Key>
(maybe more than 5 because they are recursive in theory: canDictionaryArray
contain aDictionaryArray
? can it containRunArray
?)Focused
Because SQL is statically typed, it is necessary to select function overload during planning time, so that return type is also known (the
return_type
function). Theinvoke
function needs to implement same logic again. This process should be less error-prone: once the function is bound to the query call site, its signature is known and the implementation should not need to do type checks.Performant / Efficient
It should be the compiler's / framework's job to provide vectorization, without hand-writing same logic in every function separately.
It should be the compiler's / framework's to do null checks, providing efficient tight loops for the all-non-null case without having to write such logic in every function separately.
It should be possible to write efficient functions that need thread local data structures, for example for regular expressions, without having to use thread locals and/or shared pools which introduce contention.
Describe alternatives you've considered
arrow-udf
liibrary #11413However, direct use of the library is not possible because
The library could still be part of the solution, but doesn't have to be and it's a non-goal to use a particular library.
Additional context
The text was updated successfully, but these errors were encountered: