Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow extension nodes to correctly plan physical expressions with relations #642

Closed
alamb opened this issue Jun 29, 2021 · 0 comments · Fixed by #643
Closed

Allow extension nodes to correctly plan physical expressions with relations #642

alamb opened this issue Jun 29, 2021 · 0 comments · Fixed by #643
Assignees
Labels
datafusion Changes in the datafusion crate enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Jun 29, 2021

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In IOx we have several extension nodes with Exprs which we want to compile to PhysicalExpr as part of a plan

This was previously annoying as the signature of the extension planner is:

    fn plan_extension(
        &self,
        node: &dyn UserDefinedLogicalNode,
        inputs: &[Arc<dyn ExecutionPlan>],
        ctx_state: &ExecutionContextState,
    ) -> Result<Option<Arc<dyn ExecutionPlan>>> {

Which offers no way to plan physical expressions, so I had to do something like the following workaround

            // create physical expressions in extension planner)
            let input_schema = inputs[0].schema();
            let physical_planner = DefaultPhysicalPlanner::default();
            let split_expr = physical_planner.create_physical_expr(
                stream_split.split_expr(),
                &input_schema,
                ctx_state,
            )?;

However, with the introduction of qualified names, now I have to provide a DFSchema (which should have come from the input LogicalPlan which I do not have access to at this point).

            // create physical expressions in extension planner)
            let input_schema = inputs[0].schema();
            let input_df_schema = DFSchema::try_from_qualified_schema("foo", &input_schema).unwrap();
            let physical_planner = DefaultPhysicalPlanner::default();
            let split_expr = physical_planner.create_physical_expr(
                stream_split.split_expr(),
                &input_df_schema,
                &input_schema,
                ctx_state,
            )?;

(note the "foo" needs to be the correct table name"

Describe the solution you'd like

  1. Add create_physical_expr to the PhysicalPlanner trait
  2. Pass an instance of the planner to the plan extension (rather than a ctx_state)
  3. Pass the input logical schema to each "plan extension" call as well

Basically as the internals of DataFusion evolve we also need to evolve the interfaces along with it.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@alamb alamb added the enhancement New feature or request label Jun 29, 2021
@alamb alamb self-assigned this Jun 29, 2021
@alamb alamb added the datafusion Changes in the datafusion crate label Jun 29, 2021
@alamb alamb closed this as completed in #643 Jul 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant