Query: Pay your (technical) debt on time

[smitpatel]

This is a grouping of related issues. Feel free to vote (👍) for this issue to indicate that this is an area that you think we should spend time on, but consider also voting for individual issues for things you consider especially important.

---

Creating a general tracking issue for tasks which were deferred in new query pipeline due to time constraint and also as an iteration over the design as we see customer reports we are getting.

This is a meta issue. It will evolve and add more stuff as we discuss things in team meetings (like the one we did about null semantics & command caching). I would prefer this to be assigned to me as the sticky we decided in the planning process. And we will split out smaller tasks out of this and assign to actual implementer. If you are working on something already, please add tracking issue here next to the item.

• Parameter extraction

  • Parameter-specific evaluation in ParameterExtractor and compiled queries  #19209
  • General QueryContext-based mechanism for caching of generated RelationalCommand #17598

• Preprocessor

  • Extensibility
  • Aggregation of visitors to avoid traversing whole tree multiple times
  • Nav expansion improvements
    • Processing of pending selector. When can we flow the selector through. When do we need to apply the selector to get back original shape.
    • Query: Re-evaluate concept of pending selector in nav expansion #20291
    • Processing of complex operators like GroupJoin/GroupBy from nav expansion perspective.
  • Query: Introduce an Expression for query root #20146
  • Split query for non-navigation collections #21234

• Translation

  • GroupBy aggregate translation improvement
    • Allowing more operators before aggregate
      • Distinct (also introduction of Distinct Column syntax)
      • Where using case block with null as else
      • Ignore/trim OrderBy/ThenBy since it is not useful? (wait for feedback)
    • Keep them enumerable methods or convert to queryable (enumerable methods)
  • Pending collections
    • A different way to store them to avoid disconnect
    • Multi level deep correlation predicate
    • Keep them in sync with outer
  • ColumnExpression.Table
    • Avoid storing actual Table with columnExpression
      • Different way to assign unique table alias
      • Do we ever update the table alias during any visitation?
      • readonly table alias?
      • Hash Code computation
    • Make it readonly after assigning alias?
  • Block translation of member/methodCall when store type is different from expected
    • Alternate translation? Eg. ToString
  • Block translation of collection projection when it is not possible to correlate
    • Distinct case
    • GroupBy aggregate case
  • Query: Define set of conditions for materialization on EntityShaper #20332
  • Query: Rewrite Entity Equality #20164
  • Query: Lift table from subquery when joining if no additional operations #21321
  • Generate Select * for queries when applicable for shorter SQL #21320
  • Ability to bind after client projection
  • Convert Skip/Take to RowNumber when it is more than 1 level deep #21280
  • Query: Remove additional orderings for split collection include #21273
  • Query: Add entry point in queryable method translating expression visitor #26048

• Postprocessor

  • Extensibility
  • Aggregation of visitors to avoid traversing whole tree multiple times

• ShaperCompiler

  • Aggregation of visitors to avoid traversing whole tree multiple times (pending for InMemory/Cosmos)
  • Query: Inline shapers for the cases when we don't need variables #21334
  • Query: Avoid boxing values when storing them in ResultContext #21335
  • Query: Be smart about what to store in variables for shaper #21336

• SqlProcessor

  • Aggregation of visitors to avoid traversing whole tree multiple times
  • SqlFunctionExpression nullability
    • If parameter propagates null (if arg is null then func is null)
    • If Function can be nullable (func may never be null even if arg is null)
    • API on DbFunction to configure these facets
  • SearchCondition conversion to relational in order to do conversion for bool columns with value converter
    • Opt out flag to generate shorter SQL. (non-SqlServer kind)
  • NullSemantics after parameter values are known @maumar
  • What should go after relational command caching (visitors which requires parameter values)

• Command execution

  • Relational command caching which uses lambdas rather than just parameter nullability (partial parameter sniffing) @roji
  • Disconnect one-to-one mapping between client parameters and server parameters

• Whole pipeline

  • Exception messages using core string (referenced in code with TODOs)
    • Cosmos requires Offset-Limit both
    • Move to correct assembly
  • Avoid throwing exception in places other than translation
    • Must do for GroupJoin
  • Evaluate what is being printed for exception when printing non-linq tree
  • XML Docs

[smitpatel]

Consider refactoring in cosmos similar to #21181

[smitpatel]

#21700 causes slight regression and spike in memory usage just to compute hash code. Which leads into another rabbit hole that projection is mainly comprised of column expressions coming from tables and when computing their hash code we may go inside another select expression.
Ideally this is not necessary

• The hash code computation of SelectExpression will figure out mismatch Tables through SelectExpression.Tables
• If graph is consistent then a particular column expression would only come from a particular table.
  I tried to apply solution that ColumnExpression can be compared with just reference but it caused test failure.
  A solution where column expression are created from a fixed place of reference could resolve the issue as no 2 similar column expression instance would be generated. Things to keep in mind for that is all the places which can create column expression should not create one if there is already generated one. MakeNullable would be a bottleneck.