Improved UDF/UDAF interfaces

[miguno]

Background

KSQL supports UDFs and UDAFs today. However, there are certain limitations that limit their usefulness in practice, especially for UDAFs. This ticket is about addressing the most commonly requested limitations.

UDF limitations

• The number 1 feature request is supporting more complex data types (map/list/struct), including the ability to create fields of those types.
  • KSQL UDF doesn't support returning multiple output values #1819
  • UDF does not support List as a return type #1791
  • Problem with UDFs returning Map? #525
• The number 2 feature request (a close second) is supporting 'varargs' functions.
  • UDFs should support Varargs. #1509

Related:

• Implementations of UDF for different parameterizations of Map or List are not distinguished #2085
• UDFs should support casting of appropriate number types to match function definition when there isn't an exact match #1581

UDAF limitations

For UDAFs in particular are currently very limited, and we should consider prioritizing work on better UDAFs over UDFs.

• A UDAF accepts only a single input parameter.
  • Add the ability to use generic or wildcard type parameters and return types.
  • Allow for varargs inputs (possibly less urgent for UDAF than for UDF)
• You can only store a single field in-between UDAF invocations, and that field must also have the same data type as the final result. This makes it impractical to implement even a simple AVERAGE function because one cannot store both running_total and record_count values.

[miguno]

cc @blueedgenick

[ankitchiplunkar]

Would really appreciate increased functionality in UDAF's to perform basic operations such as mean, standard deviation etc.

In comparison here is a sample UDAF of Spark to calculate the geometric mean. https://docs.databricks.com/spark/latest/spark-sql/udaf-scala.html It has two extra features which enable calculation of mean/standard_deviation for aggregates

1. bufferSchema: Used to store multiple fields during UDAF invocation
2. evaluate() : Method to finally convert the bufferSchema to return value.

[miguno]

Thanks for the feedback @ankitchiplunkar !

If you don't mind, could you also give your upvote to this feature request by giving a 👍 reaction to the first message in this thread? This allows us to automatically track number of upvotes per feature.

In addition to what you said above, which specific UDAFs are you interested in? Beyond allowing users to provide their own (and making it easier to implement these, which is what this feature request covers), I am asking because some commonly requested UDAFs we can also support out of the box.

[ankitchiplunkar]

@miguno we wanted to perform basic stats on a grouped data stream and std deviation is the first use case. We have just open-sourced a UDF which performs basic math operations and inturn enables to calculate the standard deviation.
Would appreciate your inputs
https://github.com/tokenanalyst/ksql-math-udf
https://twitter.com/thetokenanalyst/status/1065597747993165824

[miguno]

@ankitchiplunkar: Feel free to send us a PR to contribute any such UDFs to this repo, so they are included out of the box.

[ankitchiplunkar]

@miguno for sure, would create a PR containing these UDF's

[blueedgenick]

for the sake of being explicit, we should call out here that UDAFs would also greatly benefit from some of the capabilities called out for UDFs in the issue description above.
Specifically:

• the ability to use generic or wildcard type parameters and return types
• varargs inputs (though this one might be less urgent for UDAF than for UDF)

[blueedgenick]

one more that we somehow missed off the original list:

• make ProcessorContext available inside of an UDF implementation

[peoplemerge]

+1

[kewaltyagi]

+1

[agavra]

🎉 I believe these are all addressed in the upcoming release of KSQL, and more! Thanks to a huge team effort with @vpapavas and @purplefox contributing some features around this as well.

I'm going to close out this ticket as it's been mostly addressed, but feel free to open any specific more targeted feature requests/bug reports if you encounter them!