Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show optimized physical and logical plans in EXPLAIN #744

Merged
merged 3 commits into from
Jul 19, 2021

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jul 18, 2021

Which issue does this PR close?

Closes #221

Rationale for this change

"The EXPLAIN command is useful to show the plan of the query engine. However, the current output shows the un-optimized logical plan, which is not very useful as it maps almost 1:1 to the query."

Most users issue an EXPLAIN command to see what the query engine is actually going to run.

What changes are included in this PR?

Change EXPLAIN to show the optimized logical plan (after all logical optimizer passes) and optimized physical plan (after all physical optimizer passes).

Working with sql.rs reminded me of my goal for cleaning up sql tests, and so I filed #743

Are there any user-facing changes?

Explain plan output is different

For example, before this PR

> EXPLAIN SELECT c2 from foo;
+--------------+----------------------------------+
| plan_type    | plan                             |
+--------------+----------------------------------+
| logical_plan | Projection: #foo.c2              |
|              |   TableScan: foo projection=None |
+--------------+----------------------------------+
1 row in set. Query took 0.002 seconds.

and after this PR (note the inclusion of the projection pushdown and physical plan)

> EXPLAIN SELECT c2 from foo;
+---------------+--------------------------------------------------------------------------+
| plan_type     | plan                                                                     |
+---------------+--------------------------------------------------------------------------+
| logical_plan  | Projection: #foo.c2                                                      |
|               |   TableScan: foo projection=Some([1])                                    |
| physical_plan | ProjectionExec: expr=[c2@0 as c2]                                        |
|               |   RepartitionExec: partitioning=RoundRobinBatch(16)                      |
|               |     CsvExec: source=Path(/tmp/foo.csv: [/tmp/foo.csv]), has_header=false |
+---------------+--------------------------------------------------------------------------+

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Jul 18, 2021
@@ -749,32 +754,9 @@ impl DefaultPhysicalPlanner {
"Unsupported logical plan: CreateExternalTable".to_string(),
))
}
LogicalPlan::Explain {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously the physical planner could handle an LogicalPlan::Explain anywhere in the tree. However, I can't think if cases where this can actually occur and it made adding the final physical plan challenging.

So I have changed the code to special case handling Explain as the root.

@@ -47,6 +47,41 @@ use datafusion::{
};
use datafusion::{execution::context::ExecutionContext, physical_plan::displayable};

/// Copied from datafusion/test/mod.rs
/// TODO: consolidate
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically a copy/paste from test/mod.rs -- I have plans to fix this in #743

@alamb
Copy link
Contributor Author

alamb commented Jul 18, 2021

FYI @Dandandan

@jimexist
Copy link
Member

related #752 and #751 cc @alamb

@alamb
Copy link
Contributor Author

alamb commented Jul 19, 2021

FYI @Dandandan since you requested the original feature

@alamb alamb merged commit 3fb600d into apache:master Jul 19, 2021
@alamb alamb deleted the alamb/fix_explain_plans branch July 19, 2021 16:45
@Dandandan
Copy link
Contributor

Thanks, looks great! @alamb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The non-verbose EXPLAIN statement doesn't show optimized logical plan
3 participants