Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Expr.cast() to accept python types #753

Closed
Tracked by #828
timsaucer opened this issue Jul 13, 2024 · 0 comments · Fixed by #858
Closed
Tracked by #828

Enhance Expr.cast() to accept python types #753

timsaucer opened this issue Jul 13, 2024 · 0 comments · Fixed by #858
Labels
enhancement New feature or request

Comments

@timsaucer
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
To make a more user friendly interface, Expr.cast() should accept a python type and attempt to convert it to the appropriate pyarrow data type. This is predicated upon pull request #750 being merged.

Describe the solution you'd like
See the below example from @datapythonista

Describe alternatives you've considered
Alternative is to leave as is, which is operable.

Additional context
This example of enhancement requests include the desired use case:

import datafusion
from datafusion import col, lit, functions as f
import pyarrow


# something like this would be implemented internally, so users can call `datafusion.read_*`
def _read_parquet(*args, **kwargs):
    ctx = datafusion.SessionContext()
    return ctx.read_parquet(*args, **kwargs)
datafusion.read_parquet = _read_parquet  # creating an alias of `read_*` functions so users don't need to know about `SessionContext` when the defaults are fine


df = (datafusion.read_parquet("buildings.parquet")
                .filter(  # `.filter()` accepting multiple conditions (which will be an AND) instead of having to use `&` with its operator precedence problems
                    col("is_offplan") == False,
                    col("rooms") >= 2,  # `.lit(2)` not being required, and Python literals working with operators
                )
                .aggregate(
                    [col("area_name_en")],
                    [f.mean(col("has_parking").cast(float))],  # `.cast()` accepting Python types, which would be internally converted to the PyArrow equivalent
                )
                .select(
                    col("area_name_en").alias("Area"),
                    col("AVG(has_parking)").alias("Percentage of buildings with parking"),  # removing the default `?table?` in column names, the column name was "AVG(?table?.has_parking)"
                )
     )
@timsaucer timsaucer added the enhancement New feature or request label Jul 13, 2024
mesejo added a commit to mesejo/arrow-datafusion-python that referenced this issue Sep 6, 2024
mesejo added a commit to mesejo/arrow-datafusion-python that referenced this issue Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant