Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions.transform #811

Open
timsaucer opened this issue Aug 17, 2024 · 0 comments
Open

Add functions.transform #811

timsaucer opened this issue Aug 17, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@timsaucer
Copy link
Contributor

timsaucer commented Aug 17, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

This is related to #807 but slightly different. As a user I want to be able to apply a transform function to every element of an array. Suppose I have a dataframe where one column contains an array.

DataFrame()
+-----------+
| a         |
+-----------+
| [1, 2, 3] |
| [4, 5]    |
| [6]       |
| []        |
+-----------+

I might want to convert this with a single line such that each element is converted to whether it is odd.

DataFrame()
+---------------------+
| a_odd               |
+---------------------+
| [True, False, True] |
| [False, True]       |
| [False]             |
| []                  |
+---------------------+

It would be great to do something like datafusion.functions.transform(col("a"), my_func).alias("a_odd")

Describe the solution you'd like

Right now I'm not completely sure how I would do this. I think I might have to do an unnest, transform, and aggregate.

Describe alternatives you've considered

These could be done with a UDF.

Additional context

@timsaucer timsaucer added the enhancement New feature or request label Aug 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant