Implements OUTER bag operators #400

alancai98 · 2023-06-27T01:19:52Z

Implements the OUTER bag operators. Corrects a bug with the parsed set quantifier for bag operators. Previously when a set quantifier was unspecified, the default was ALL. Changed to be DISTINCT.

Other changes from this PR

BREAKING: partiql-ast: rename of SetExpr to BagOpExpr and SetOp to BagOp
- Affects the AST and visitor
BREAKING: partiql-parser: Parsed struct's ast field is now an ast::AstNode<ast::TopLevelQuery>
BREAKING: partiql-eval: Evaluable trait's update_input fn now also takes in an EvalContext
Closes Decide if branch_num is needed in logical plan #237. branch_num is now used for bag operators. Removed the comment for add_flow_with_branch_num fn.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

CHANGELOG.md

codecov · 2023-06-27T01:29:56Z

Codecov Report

Patch coverage: 89.83% and project coverage change: +0.24 🎉

Comparison is base (bb3487b) 81.16% compared to head (10745d3) 81.41%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #400      +/-   ##
==========================================
+ Coverage   81.16%   81.41%   +0.24%     
==========================================
  Files          56       56              
  Lines       14753    14912     +159     
  Branches    14753    14912     +159     
==========================================
+ Hits        11974    12140     +166     
+ Misses       2260     2251       -9     
- Partials      519      521       +2

Impacted Files	Coverage Δ
partiql-ast/src/ast.rs	`1.06% <ø> (ø)`
partiql-eval/src/error.rs	`28.57% <0.00%> (ø)`
partiql-eval/src/eval/mod.rs	`72.09% <0.00%> (ø)`
partiql-logical/src/lib.rs	`40.80% <0.00%> (-0.97%)`	⬇️
partiql-parser/src/lib.rs	`86.95% <ø> (ø)`
partiql-logical-planner/src/lib.rs	`94.31% <80.00%> (+5.55%)`	⬆️
partiql-eval/src/eval/evaluable.rs	`89.46% <87.93%> (-0.11%)`	⬇️
partiql-logical-planner/src/lower.rs	`85.78% <95.83%> (+0.90%)`	⬆️
partiql-ast-passes/src/name_resolver.rs	`82.69% <100.00%> (ø)`
partiql-ast-passes/src/partiql_typer.rs	`61.25% <100.00%> (+0.31%)`	⬆️
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

github-actions · 2023-06-27T01:32:38Z

Conformance comparison report

	Base (`bb3487b`)	`3dbde28`	+/-
% Passing	83.35%	84.12%	0.76%
✅ Passing	5268	5318	50
❌ Failing	1052	1004	-48
🔶 Ignored	0	0	0
Total Tests	6320	6322	2

Number passing in both: 5268

Number failing in both: 1004

Number passing in Base (bb3487b) but now fail: 0

Number failing in Base (bb3487b) but now pass: 48

The following test(s) were previously failing but now pass. Before merging, confirm they are intended to pass:

Click here to see

partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_with_order_by_limit_offset_on_children
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_coerce_null_missing
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_6_value_coercion_coercion_of_single_value
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_intersect_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_all
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_coerce_null_missing
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_except_all
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_intersect_with_order_by_limit_offset_on_children
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_coerce_list
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_7_select_fr_o_m_engineering_employees_outer_except
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_2_3_union_of_compatible_relations_mismatch_column_names_using_outer_union
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_intersect_all
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_coerce_scalar
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_3_outer_union_of_heterogenous_relations
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_intersect_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_except_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_6_value_coercion
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_6_value_coercion_coercion_of_single_value
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_7_select_fr_o_m_engineering_employees_outer_except
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_intersect_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_7_result_is_the_empty_bag
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_intersect_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_except_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_with_order_by_limit_offset_on_children
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_all
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_except_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_2_3_union_of_compatible_relations_mismatch_column_names_using_outer_union
partiql_tests::eval_equiv::rfc::_0007::_0007::bag::strict_example_6_value_coercion
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_coerce_list
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_coerce_struct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_with_order_by_limit_on_children_and_bag_op
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_3_outer_union_of_heterogenous_relations
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_union_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_except_distinct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_except_with_order_by_limit_offset_on_children
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_7_engineering_employees_outer_union_missing
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_coerce_struct
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_except_all
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_7_engineering_employees_outer_union_missing
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_union_coerce_scalar
partiql_tests::eval::rfc::_0007::_0007::bag::permissive_example_6_value_coercion
partiql_tests::eval::rfc::_0007::_0007::bag::strict_example_7_result_is_the_empty_bag
partiql_tests::eval_equiv::rfc::_0007::_0007::bag::permissive_example_6_value_coercion
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::permissive_outer_intersect_with_order_by_limit_offset_on_children
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_except_with_order_by_limit_offset_on_children
partiql_tests::eval::primitives::operators::operators::bag_operators::bag_operators::strict_outer_intersect_all

partiql-eval/src/eval/evaluable.rs

and further testing need addressed

jpschorr

I proposed some clean-up via commit that I pushed. Everything else LGTM.

alancai98 · 2023-07-11T23:33:46Z

I proposed some clean-up via commit that I pushed. Everything else LGTM.

Thanks, clean-up changes in 10745d3 (e.g. using itertools, iterating over ValueIntoIterator rather than Bag, and multiplicity cleanup) look good to me.

alancai98 self-assigned this Jun 27, 2023

alancai98 commented Jun 27, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

alancai98 force-pushed the feat-bag-ops branch 2 times, most recently from 7dcf855 to eb33c41 Compare June 29, 2023 20:55

alancai98 changed the title ~~[WIP] Outer bag op implementation~~ Outer bag op implementation Jun 29, 2023

alancai98 changed the title ~~Outer bag op implementation~~ Implements OUTER bag operators Jun 29, 2023

alancai98 force-pushed the feat-bag-ops branch 2 times, most recently from 4fe4a97 to a8e8974 Compare June 30, 2023 00:04

alancai98 marked this pull request as ready for review June 30, 2023 00:06

alancai98 requested review from jpschorr and am357 June 30, 2023 00:18

alancai98 force-pushed the feat-bag-ops branch 2 times, most recently from 622e861 to 2b73f21 Compare July 10, 2023 18:12

alancai98 mentioned this pull request Jul 10, 2023

Add subquery lowering #408

Open

jpschorr reviewed Jul 11, 2023

View reviewed changes

partiql-eval/src/eval/evaluable.rs Outdated Show resolved Hide resolved

alancai98 added 6 commits July 11, 2023 15:33

Initial (outer) bag op implementation

c27d8fc

Rebase following #401; change to use new TopLevelQuery; still TODOs

cc60f98

and further testing need addressed

Add back order_by, limit_offset to QueryClauses

ad06bf8

Update partiql-tests, address TODOs, update changelog

dde5cc9

Remove comment about branch_num

50eb72b

Fix outer bag intersect

d1556dc

alancai98 force-pushed the feat-bag-ops branch from e0bfe70 to d1556dc Compare July 11, 2023 22:44

alancai98 requested a review from jpschorr July 11, 2023 22:53

Update submodule with another outer intersect test

bc4196f

alancai98 force-pushed the feat-bag-ops branch from 93f5f2f to bc4196f Compare July 11, 2023 23:00

Update Bag operators to use iteration rather than temporary Bags

10745d3

jpschorr approved these changes Jul 11, 2023

View reviewed changes

alancai98 merged commit d1bdfb0 into main Jul 12, 2023

alancai98 deleted the feat-bag-ops branch July 12, 2023 17:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implements OUTER bag operators #400

Implements OUTER bag operators #400

alancai98 commented Jun 27, 2023 •

edited

Loading

codecov bot commented Jun 27, 2023 •

edited

Loading

github-actions bot commented Jun 27, 2023 •

edited

Loading

jpschorr left a comment

alancai98 commented Jul 11, 2023

Implements OUTER bag operators #400

Implements OUTER bag operators #400

Conversation

alancai98 commented Jun 27, 2023 • edited Loading

codecov bot commented Jun 27, 2023 • edited Loading

Codecov Report

github-actions bot commented Jun 27, 2023 • edited Loading

Conformance comparison report

jpschorr left a comment

Choose a reason for hiding this comment

alancai98 commented Jul 11, 2023

alancai98 commented Jun 27, 2023 •

edited

Loading

codecov bot commented Jun 27, 2023 •

edited

Loading

github-actions bot commented Jun 27, 2023 •

edited

Loading