Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add SubqueryAliasRel #648

Closed

Conversation

Blizzara
Copy link
Contributor

Add SubqueryAliasRel that can be used to name/(re)qualify a downstream relation.

As Substrait mostly only referes to columns by their index, there is no inherent need for table name/qualifiers within Substrait. However, some consumers, e.g. DataFusion, require column names to be either unique or qualified for joins, which is troublesome w/o the possibility to qualify relations.

@Blizzara
Copy link
Contributor Author

In my usecase, this is mainly a problem with VirtualTables in DataFusion. An alternative would be to add a name into the VirtualTable (or ReadRel) directly. However, having this SubqueryAliasRel would be more general.

FWIW, both Spark and DataFusion have some version of SubqueryAlias: Spark, DataFusion

@EpsilonPrime
Copy link
Member

For Substrait, the answer might be to automatically name/number the tables in order to provide that distinction. Ignoring how the tables are numbered for a bit, just being able to say column X from table Y should be enough to solve the naming problem without actually naming.

A number of ways to number are possible including just referring to the location in the plan (plan id in Spark), registering the read relations in some discovery order, or using a URN for the local file/virtual table so that table references could be coalesced despite being read multiple times.

Technically I would not call this a subquery alias we aren't aliasing the subtree but instead the fields at a point in the subtree.

message SubqueryAliasRel {
RelCommon common = 1;
Rel input = 2;
repeated string names = 3;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have structured content would these names be in depth first form as described elsewhere? If so, that comment should be copied here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be aliaising the subquery, not the field names - so this is more similar to the names on a NamedTable rather than the depth-first order used for fields.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on that it sounds like this is a list of names that all refer to the input relation.

@Blizzara
Copy link
Contributor Author

Technically I would not call this a subquery alias we aren't aliasing the subtree but instead the fields at a point in the subtree.

The proposal here was to alias the subtree specifically, ie. do the same as what Spark's and DataFusion's SubqueryAlias does.

For Substrait, the answer might be to automatically name/number the tables in order to provide that distinction. Ignoring how the tables are numbered for a bit, just being able to say column X from table Y should be enough to solve the naming problem without actually naming.

Yes, Substrait itself does not require this addition - the Substrait plan is valid w/o it, as columns are referred by indices. However it is not possible (or at least I couldn't figure out how, yet) to do lossless roundtrips from DataFusion -> Substrait -> DataFusion for some cases without this. Like for example:
SELECT * FROM (VALUES (1, 'a')) AS t1(c1,c2) JOIN (VALUES (1, 'b')) AS t2(c3,c4) ON t1.c1 == t2.c3
or
SELECT p1.a p1_a, p2.a p2_a FROM data p1, data p2

I don't know how much value that argument holds, but I've found the roundtrip tests super useful when developing Substrait functionality for DataFusion and Spark - makes it much easier to confirm correctness.

A number of ways to number are possible including just referring to the location in the plan (plan id in Spark), registering the read relations in some discovery order, or using a URN for the local file/virtual table so that table references could be coalesced despite being read multiple times.

Yeah, there are ways. But it will make the roundtrip testing more difficult (since now the Substrait consumer has different logic for naming things than the original producer of the query plan), and may make debugging more difficult for users if the tables are not identifiable by human-written ids but only locations or generated ids. Again, not sure how much weight that argument holds - depends on what the priorities are, I guess :)

@Blizzara
Copy link
Contributor Author

Actually, when I say "roundtrip", it's a bit more than that - w/o this, I think it's also impossible to maintain output schema for the roundtrip, as the columns after X -> Substrait -> X conversions would have different qualifiers than at the beginning. That is also not crucial for correctness, but helps with "quality of life", I think.

@EpsilonPrime
Copy link
Member

Actually, when I say "roundtrip", it's a bit more than that - w/o this, I think it's also impossible to maintain output schema for the roundtrip, as the columns after X -> Substrait -> X conversions would have different qualifiers than at the beginning. That is also not crucial for correctness, but helps with "quality of life", I think.

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

@Blizzara
Copy link
Contributor Author

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

True. But that feels less natural and requires one to use something else to generate the original substrait (or write it manually). There are cases where it makes sense, to ensure interoperability, but I'm not sure if it's feasible to be the main way for testing.

Also, it doesn't help with the human understandability/debuggability of the plans created from Substrait, which I think would/will be important.

Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thank you for taking this on. Many people have asked for some kind of capability like this. I think it's important.

Is this meant to be in included together with #649 or are they mutually exclusive? If they are meant to be included together then why couldn't you apply a rename by specifying the rel_names field on the subquery relation?

Next question. Are we working with subquery, the expression, or are we working with subquery, the physical operator? Or, in datafusion terms, are you hoping to rename Expr::ScalarSubquery or LogicalPlan::Subquery.

Judging from the existence of LogicalPlan::SubqueryAlias I am assuming you are working with the latter. However, substrate has no concept of a "subquery physical operator". Can you describe a little bit how that operator works? Can a LogicalPlan::Subquery output more than a single column? If so, does this relation rename all of the output columns?

@@ -144,6 +144,14 @@ message ReadRel {
}
}

// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
message SubqueryAliasRel {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this unique to subqueries? Wouldn't you want to be able to insert an alias at any point in a plan?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's just the name Spark and DataFusion uses - it can be added anywhere in the plan

@@ -144,6 +144,14 @@ message ReadRel {
}
}

// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
message SubqueryAliasRel {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When adding a new relation you should add it to the Rel one of or else it will not be usable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@westonpace
Copy link
Member

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

I do think the X -> Substrait -> X path is an important one. I think there are going to be many cases where people are using Substrait internally and they feel comfortable relying the fact that the consumer / producer are part of the "same solution" and can thus, for example, rely on the same extensions. I think this would mean that information that survives X -> Substrait -> X is valid even if it wouldn't survive X -> Substrait -> Y -> Substrait -> X (because Y wouldn't know how to interpret the field).

Copy link
Contributor Author

@Blizzara Blizzara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thank you for taking this on. Many people have asked for some kind of capability like this. I think it's important.

Is this meant to be in included together with #649 or are they mutually exclusive? If they are meant to be included together then why couldn't you apply a rename by specifying the rel_names field on the subquery relation?

Mutually exclusive - both independently provides the needed functionality, just in different ways. I added the other one for discussion to decide which one makes more sense (if either) :)

Next question. Are we working with subquery, the expression, or are we working with subquery, the physical operator? Or, in datafusion terms, are you hoping to rename Expr::ScalarSubquery or LogicalPlan::Subquery.

Latter - this would be turned into a LogicalPlan::SubqueryAlias (or its equivalent in Spark)

Judging from the existence of LogicalPlan::SubqueryAlias I am assuming you are working with the latter. However, substrate has no concept of a "subquery physical operator". Can you describe a little bit how that operator works? Can a LogicalPlan::Subquery output more than a single column? If so, does this relation rename all of the output columns?

Yeah, the "subquery" is maybe a misnomer here. The LogicalPlan::SubqueryAlias outputs the same columns as its input, but with the qualifiers of the names changed to the new qualifier/alias provided in the SubqueryAlias.

So as an example, if at the beginning of the query you have a table called "tbl" with columns "col1" and "col2", then you can refer to those columns either with unqualified names ("col1", "col2") or qualified names ("tbl.col1", "tbl.col2"). If you then wrap that plan in a SubqueryAlias(alias = "something_else"), you can then refer to the columns using the new qualified names ("something_else.col1", "something_else.col2") (or still the unqualified names). Where this gets important is with joins, where one may need the qualified names to distinguish between columns coming from left or right.

@@ -144,6 +144,14 @@ message ReadRel {
}
}

// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
message SubqueryAliasRel {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -144,6 +144,14 @@ message ReadRel {
}
}

// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
message SubqueryAliasRel {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's just the name Spark and DataFusion uses - it can be added anywhere in the plan

@westonpace
Copy link
Member

Mutually exclusive - both independently provides the needed functionality, just in different ways. I added the other one for discussion to decide which one makes more sense (if either) :)

If we go with the other approach (metadata on RelCommon) do we still need some kind of "no-op" node that can be used to alias a relation tree? Or would you envision that we just map DF Subquery -> SubqueryAlias to a single node (subquery) that has alias metadata?

Also, are you wanting to add a node for the "subquery physical operator"? (e.g. LogicalPlan::Subquery). That would be a different PR, I'm just curious.

@Blizzara
Copy link
Contributor Author

If we go with the other approach (metadata on RelCommon) do we still need some kind of "no-op" node that can be used to alias a relation tree? Or would you envision that we just map DF Subquery -> SubqueryAlias to a single node (subquery) that has alias metadata?

In that case I don't think there'd be need to add "no-op" node. Producers could map their "SubqueryAlias" by adding metadata to the preceding Rel node, and consumers could read the metadata (if they want to) and add a "SubqueryAlias" to wrap the Rel.

I kinda like that approach since it makes it clear the metadata/alias is optional, and consumers/producers don't need to care about it at all unless they want to. Contrast to this SubqueryAliasRel where consumers would need to start handling the node (even if just to pass through the input rel).

Also, are you wanting to add a node for the "subquery physical operator"? (e.g. LogicalPlan::Subquery). That would be a different PR, I'm just curious.

I haven't seen the need for that, yet at least, so not planning on it currently.

@vbarua
Copy link
Member

vbarua commented Jun 12, 2024

I kinda like that approach since it makes it clear the metadata/alias is optional, and consumers/producers don't need to care about it at all unless they want to. Contrast to this SubqueryAliasRel where consumers would need to start handling the node (even if just to pass through the input rel).

I was going to say something similar, but you said it better than I could myself. This is the reason I'm in favour of the solution you're proposing in #649.

@Blizzara
Copy link
Contributor Author

Closing in favor of #649 - that approach seems like the better path forward overall, so let's focus discussion there!

@Blizzara Blizzara closed this Jun 19, 2024
EpsilonPrime pushed a commit that referenced this pull request Jul 3, 2024
Same goal as in #648 - to support what Spark's and DataFusion's
SubqueryAlias relation does.

As Substrait mostly only referes to columns by their index, there is no
inherent need for table name/qualifiers within Substrait. However, some
consumers, e.g. DataFusion, require column names to be either unique or
qualified for joins, which is troublesome w/o the possibility to qualify
relations.

Also, for debugging failed plans and for roundtrip testing of X ->
Substrait -> X conversions, it would be convenient to have proper,
human-readable names to refer to.

Closes #571
@Blizzara Blizzara deleted the avo/add-subquery-alias-rel branch July 8, 2024 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants