Extend expression trees to cover more/all language constructs

[gafter]

Expression trees today support a limited subset of expression lambdas. This issue is to track the request to extend them to support more or all of the language constructs that can be expressed in a lambda.

A prototype for work in this direction can be found on https://github.com/bartdesmet/ExpressionFutures/tree/master/CSharpExpressions and https://github.com/bartdesmet/roslyn/tree/ExpressionTrees (based on a fork of Roslyn a while ago).

/cc @bartdesmet

[bbarry]

previous discussion: https://github.com/dotnet/roslyn/issues/2060

notable things missing:

• null-coalescing operators such as a?.b, a?[b]
• support for dynamic
• support for await
• default parameters
• named parameters
• base access
• assignment
• multi-dim array initialization
• dictionary initialization
• increment/decrement
• statements
• unsafe code
• out and ref parameters
• method group references
• ref returns
• tuples
• throw expression
• patterns

[NetMage]

Adding comment from issue #503:

Expression Trees should be extended to support the C# 6.0 null propagating operator so that it can be used with Linq, with a translation to allow providers such as Linq to SQL to add support.

The discussion on CodePlex doesn't seemed to have moved over to here, and has been sitting around for 3 years - I think it is worth revisiting, and ideally implementing. An ideal use case is an Outer Join in Linq to Objects.

https://roslyn.codeplex.com/discussions/571077

[stevozilik]

+1
Not having Null-propagating operator ?. in Expression Trees is a major blocker for unit-testing left outer joins

[stakx]

I'd like to add my vote for this proposal for the sake of the Moq 4.x mocking library, whose API relies heavily on expression tree lambdas. In particular, support for the following two language features in expression tree lambda would be hugely beneficial to Moq. We could then fix the majority of currently reported bugs, and add some long requested features:

1. assignments
2. ref returns — @bbarry, perhaps you could add this language feature to your list above?

I'm happy to further explain my reasons for prioritising these two, but don't want to go off on a tangent unbidden.

[olmobrutall]

With the (awesome) addition of nullable reference types in C# 8.0 having null-propagation operation in expression trees goes from "very important" to "you almost can not use Entity Framework without this feature".

Looking forward for knowing the design details of how this could be implemented in order to adapt my LINQ provider.

[alrz]

notable things missing:

I wonder if local functions could be translated to SQL CTEs.

[stakx]

Bump.

@gafter, I was wondering whether there's any news from the C# language team about this. I understand that it's probably more interesting to design and plan brand-new language features like Span<T> etc. than to maintain an established one, but at the same time we have this amazing feature -- code as data, in the form of expression trees -- that gets totally left behind. C# is now at version 7.1, while expression trees are still at approx. C# 4 or 5. The gap keeps widening, and the relative limitations of expression trees therefore become more obvious and more frustrating.

Is there any chance this can be given a higher priority? Is there anything the community can do to help advance expression trees?

[scalablecory]

Async/await support is probably the largest undertaking on that list, but I'd love it. It would significantly help projects performing complex code-gen.

[stakx]

@scalablecory - I totally agree this would be nice. That being said, I question the likeihood of getting async / await support in expression trees when even fundamental and much more basic language features such as = (assignment) aren't supported. 😢

I think it would be awesome if we could just get the ball rolling, i. e. we could get agreement from the C# language team and/or the Roslyn team to look at the most basic missing features for a start. (I'd be happy to help in any way I reasonably can.) If we manage to get there, perhaps the more complex features will eventually follow, too.

[gafter]

It was discussed briefly on Monday... noting for example that we currently do not have support for ?. and friends, and that will become more annoying as we add support for non-nullable reference types.

/cc @dotnet/ldm It would help if we had a member of the LDM who championed this (even though it isn't technically a language feature). Any takers?

[bbarry]

I think there would need to be a language change to support statements in expression trees because expression tree types explicitly states:

• Not all lambda expressions can be converted to expression trees. For instance, lambda expressions with statement bodies, and lambda expressions containing assignment expressions cannot be represented. In these cases, a conversion still exists, but will fail at compile-time. These exceptions are detailed in Anonymous function conversions.

emphasis added.

For everything else:

the precise rules for constructing an expression tree when a lambda expressionis converted to an expression tree type, are both outside the scope of this specification.

Should it be (libraries that walk these often assume specific patterns; the fact that according to the language spec they appear to be able to change between compiler versions is surprising isn't it)?

(OT, it looks like there are numerous spaces missing from this document after the word "expression"; are PRs being accepted for these sorts of things yet?)

[gafter]

@bbarry PRs to fix typos in the spec are most welcome.

[olmobrutall]

@gafer Is there any issue where the changes on the expression API are being discussed?

[gafter]

Not that I am aware

[olmobrutall]

I don’t see ?. in expression trees anywhere in the selected features for c# 8.

Any chance it will be included? Is considered a part of nullable reference types?

I had three bugs in production because of SQL implicit nulls in left joins...

What can we do to help this move forward?

[Igorbek]

Also, expression tree-like types proposal seems awesome:

• https://github.com/bartdesmet/roslyn/blob/ExpressionTreeLikeTypes/docs/features/expression-types.md
• https://github.com/bartdesmet/roslyn/tree/ExpressionTreeLikeTypes

[sakno]

I would like to propose ready-to-use extensions of Expression Trees presented in standard library:

• AwaitExpression is a node type inherited from System.Linq.Expressions.Expression and represents arbitrary await expression inside of async lambda. Explanation here.
• ForEachExpression allows to construct foreach loop expression
• ForExpression allows to construct for loop
• InterpolationExpression is a string interpolation expression introduced in C# 6
• UsingExpression allows to write using code blocks.
• LockExpression represents lock statement in C#
• WhileExpression allows to construct while or do-while loop

This implementation can be used as a proof-of-concept for await and other expression types. Expression tree rewriter based on Lowering technique is located here. Transformation algorithm is similar to Roslyn but differs in handling of try-catch blocks. Supported return types from async lambda are Task, Task<R>, ValueTask, ValueTask<R>.

[AartBluestoke]

according to https://github.com/dotnet/corefx/issues/33170 System.Linq.Expressions is archived meaning:

When archived, we would not accept features contributions to the packages. We would consider changes that address significant bugs or regressions, or changes that are necessary to continue shipping the binaries. Other changes will be rejected.

[AartBluestoke]

https://github.com/dotnet/corefx/issues/32523 (Expression trees should support Span, stackalloc, ref struct ) was closed as "wont-fix" because Linq.Expressions is archived

[orthoxerox]

I guess no one wants to maintain what is basically another version of the compiler, but at the moment no one on the LDT is willing to champion and prioritize a different solution for runtime code generation over other features either.

[zspitz]

@orthoxerox Couldn't Roslyn itself be used for runtime code generation? Except that AFAICT Roslyn cannot be run in a single-threaded environment.

[olmobrutall]

https://github.com/dotnet/csharplang/blob/main/meetings/2018/LDM-2018-06-06.md

For now, let's mull it over for a couple of months, pursue information, crisp up the plan. Only after a while will we be able to free up compiler resources.

That was 50 months ago :)

[CyrusNajmabadi]

Code generators only came into existence in the last two years, and are held to same general roslyn standard that this API develops at the very least at the pace of the language.

[bachratyg]

@olmobrutall

In order to make the transition process as easy as possible, something like following helper classes

I believe this functionality is already available via Expression.Reduce

[bachratyg]

Code generators only came into existence in the last two years, and are held to same general roslyn standard that this API develops at the very least at the pace of the language.

So if a code generator encounters an unknown construct in the code that it's not prepared to handle (e.g. required) it would generate bad code or simply crash?

[CyrusNajmabadi]

So if a code generator encounters an unknown construct in the code that it's not prepared to handle (e.g. required) it would generate bad code or simply crash?

IMO, it should emit a diagnostic stating that it is not compatible with this language construct.

[olmobrutall]

I believe this functionality is already available via Expression.Reduce

AFAIK, Expression.Reduce is for lowering custom expression nodes to standard ones, for example before calling .Compile().

I don’t think Expression.Coalesce reduces to a Expression.Condition, so I don’t think Expression.NullPropagation should either.

Instead Expression.NullPropagation should compile to IL just fine out of the box

[stakx]

Given the discussion above it doesn't seem very probable that the present System.Linq.Expressions trees will simply get extended with additional node types. I'm wondering what some alternative options might be.

Creating a totally new expression tree abstraction whose design somehow avoids the extensibility issues of what we're dealing here? (What would be the point of abandoning SLE only to end up in the exact same situation in a few years' time, only the compiler team would then have two legacy expression tree models instead of one?) What would such a radical re-design have to look like?

Or – I'm sure it's been discussed already, but I can't remember if & where I saw that idea – wouldn't it be possible to introduce an Roslyn extensibility point akin to source generators, which would allow user-code libraries to provide compile-time "code quotation factories" that turn a Roslyn AST of type D (where D is any delegate type) into an AST of type E<D> (where E would be any custom type linked e.g. via custom attributes to the appropriate quotation factory)? Roslyn would recognize E<D> as a code quotation type, fire up the provided factory & invoke it, and replace the original delegate expression with whatever that factory produces; basically a mechanism for user-defined implicit coercion, restricted to delegate types.

The point of this idea being to make code quotations fully customizable and thereby shifting the maintenance burden away from the compiler into libraries which can be cleanly versioned.

Those "quotation factories" could even signal (by emitting Roslyn diagnostics) what kind of expressions are supported / not supported by whatever target library the factories are associated with.

(Whether or not System.Linq.Expressions would be extracted from the compiler into an external code quotation factory lib (but still in the FCL) would remain to be seen.)

When I last thought about this idea a few years ago, I thought it would stand no chance, since it might tie the C# spec too tightly to one implementation (namely Roslyn) because of the AST model required during transformation (you wouldn't want to invent yet another code tree model, right?)... but since then source generators have come into being; would this idea be so much different from them?

[WhitWaldo]

A lot of the concern on this thread speaks to the use of expression trees in EF, but I also wanted to highlight their extensive use in the somewhat recently open-sourced Reaqtive project. This project provides a platform for hosted reactive operations allowing for stateful stateful checkpointing and recovery of standing and active subscriptions.

It provides libraries for serialization of expression trees (allowing for queries and their components to be saved in stateful storage while at rest), which is huge in itself. However, the implementation of and extendibility of the observers and observables could be made dramatically simpler if .NET had support for async expression trees, so I'd ask that these be considered in the next overhaul of expression trees, whatever shape that takes.

I understand the argument in here for breaking old dependencies on existing expression trees, but I don't necessarily see why this isn't something that is run in the .NET Core style of move-faster-likely-breaking-things as things transitioned from .NET 4 to .NET 5.

Given that the implementation of each of the proposed changes is thought to be pretty straightforward and because Bart has already done some exploration in what might be needed to support different branches of support with the ExpressionTreeLike types and even has a sample implementation of this, why can this not simply be something that is opt-in, similar to that of the nullability flag in csproj wherein developers can choose to use the legacy expression tree (selected by default) or can opt into a more modern expression tree that will continually evolve as C# does.

I don't know to what extent .NET is able to do feature flags, as it were, in the code, but what if modern expression support were an opt-in on a feature-by-feature basis? Today, my .csproj contains:

<PropertyGroup>
  <Nullable>enable</Nullable>
</PropertyGroup>

Why can't modern expression trees be similar? This would enable all available modern expression tree functionality (similar to setting language support to vNext in that it's likely something will break with new changes).

<PropertyGroup>
  <ModernExpressions>true</ModernExpressions> 
</PropertyGroup>

And especially since there's such a laundry list of individual features, allow them to be opt-in by name:

<PropertyGroup>
  <ModernExpressions>
    <NullCoaselscing>true</NullCoalescing>
    <Async>true</Async>
    <Dynamic>true</Dynamic>
  </ModernExpressions>
</PropertyGroup>

Those features that are inextricably linked to all of them are included just by enabling ModernExpressions (doesn't happen often, but does happen per the opt-in contract). But this means you don't have to enable all the new features all at once in a single release and trivially opens the door to revisiting to add new expressions features in a way that people can use if they want or simply opt-out of.

At launch, there's zero net impact until developers decide to proactively opt into the new functionality - a great many will, but if their application breaks, I suspect they'll be more reasonable about the migration. After launch, there's some impact as developers upgrade packages indiscriminately, but the analyzer saves them from themselves here. And in the longer-term, new applications get to enjoy rich expression tree support and older applications keep on using the legacy approach.

[aradalvand]

It's strange that something as important as this is not even being discussed. What's blocking this?!

[CyrusNajmabadi]

A proof of concept through at least C# 9 or so has already been created demonstrating a viable path forward ... (nullability, for example)

Nullability has absolutely no runtime impact. That's precisely why it was ok to go that route, and it's a strong reason why it could be added without being impactful to the ecosystem. That's very different from any approach that could actually end up impacting anything at runtime. :(

[WhitWaldo]

That's a good point, but at the same time, it begs the question of whether the future of C# development is just syntax updates devoid of any runtime changes for the same underlying concern of breaking some ancient backwards compatibility somewhere.

I was reading through the proof of concept last night to get a better understanding how it it's implemented and the changes to Roslyn itself don't appear to be that substantial which led me to the following. Your concern is that by changing the runtime for expressions, it could have disastrous and unknown impacts to developers consuming that functionality. The approach used in the proof of concept appears to be one that allows for over-time extensibility so as to avoid this point-in-time feature lock, but implementing that approach requires breaking the existing language runtime contract. The best path forward here then is to just leave existing expression trees exactly as they are indefinitely moving forward and chalk it up to a poor design decision in early C# days.

But then why can't you then implement the newer, extensible expression trees with the bare minimum runtime support necessary under a different namespace removed from the domain of the language team? I anticipate you saying that this just moves the needle from C# 3.0 feature lock to C# 11.0 - fair, so why not shift the actual implementation of these modern expression trees out to a separate package. The .NET library ecosystem is broad and rich and has a readily understood contract in place regarding versioning, lifecycle and modernity expectations.

The language ensures backwards compatibility to a back-breaking level - I get that, but separate implementation packages don't generally have nearly the same concern for backwards compatibility. But further, if I need what's in v1.0 and v2.0 breaks my implementation, I need only install v1.0 and compatibility is on me as the developer to ensure for a safe upgrade path.

Could this be a more acceptable path forward?

• Using a different namespace means that no changes are made to the existing implementation ensuring strict backwards compatibility
• Tweaks to the runtime are simply made in a greenfield namespace and care is taken to design this as extensibly as possible going forward to avoid the same pitfall
• Shifting the implementation to a separate package means the approach can be iterated on and shifts responsibility from the language team to the developers to ensure a package upgrade doesn't break their application in a way, while still allowing them to access and install these older versions for self-imposed feature lock

Thanks for the ongoing consideration!

[CyrusNajmabadi]

devoid of any runtime changes

Yes. We work steadfastly to not make runtime changes, except in very narrow circumstances, and when teh net benefit will likely be enormous, and highly unlikely to break anyone.

For example, we are considering changing how exceptions are thrown in async/await. This should not be noticeable at runtime, unless doing very specific groveling of the low level async/await machinery. Similarly, we may allow a params Span<T> overload to be 'better' than a params T[] overload (with the presumption that types that want to use params-span will be well-behaved and will only have different performance characteristics, not runtime semantic differences.

However, the above is very different from expression trees.

[CyrusNajmabadi]

But then why can't you then implement the newer, extensible expression trees with the bare minimum runtime support necessary under a different namespace removed from the domain of the language team?

I don't see any reason why we could not do that. I may have missed it, but did someone give a reason why that would not be possible? :)

I anticipate you saying that this just moves the needle from C# 3.0 feature lock to C# 11.0 - fair, so why not shift the actual implementation of these modern expression trees out to a separate package. The .NET library ecosystem is broad and rich and has a readily understood contract in place regarding versioning, lifecycle and modernity expectations.

I would not say this would have C# 11 lockin. Indeed, if doing this again, i would model around Roslyn, with the same promises (or more specifically, the lack thereof :)) around compat. Specifically, i would say that you should expect breaks, and that as the language revs, this API will move to match it, and it's incumbent on the consumer to deal with that (the same way we deal with roslyn changes today).

Thanks for the ongoing consideration!

Definitely :)

[WhitWaldo]

Circling back here.. if simply using a different namespace for Expressions v2 is a viable path forward for modernizing it, what are the remaining blockers, if any, to moving forward? Is there anything I can personally do to help shuffle this along apart from merely advocating for it?

[Atulin]

Found this discussion while writing yet another EF query that required me to do

.Select(x => new Dto {
    Foo = x.Foo == null ? x.Date : x.Foo
})

instead of the sane thing,

.Select(x => new Dto {
    Foo = x.Foo ?? x.Date
})

and I'm guessing I'll have to use it a while longer still

[mattwar]

That's just the coalesce operator, which has always been an expression tree node type. I'm guessing that's just a failure on the EF provider to not understand how to translate the coalesce operator into the SQL COALESCE function.

[Atulin]

This comment — dotnet/efcore#33336 (comment) — mentions that it is a limitation of expression trees.

That said... I was looking for instances of the issue in my codebase to show real-life examples, and I'm having trouble finding any. So either I hallucinated it, or it's not actually the issue I had in mind. I'll pop back into this discussion should I actually find something

[MaceWindu]

?? was never introduced to expression tree, which is probably main PITA with expression trees

[aradalvand]

Having just re-read this entire thread, all the while carefully considering the concerns and explanations provided by the likes of @CyrusNajmabadi, @333fred, et al. I must say they are making about as much sense as how many "new" C# syntactic features have had support added for them in expression trees over the last decade or so, which is, of course, a glorious zero.

Let's consider this comment by @333fred — as I think it succinctly and clearly presents the supposed justification:

The significant cost of implementing new nodes does not come from the compiler. It's fairly easy to do in the compiler. Its significant cost comes from every downstream tool now being broken because there is a new node in the tree they were not expecting, and us needing to work with EF and other consumers of expressions to create a way to signal language version support.

(emphasis mine)

Sounds reasonable, right? Except that it should take no more than 5 seconds for anyone remotely familiar with C#/.NET's and the surrounding ecosystem's progression over the last couple of years, to realize two things, which immediately obliterate this reasoning:

Swap the terms "new node" with "new built-in anything" and you'll get an equally sound argument for never adding any feature at any level in any area to the framework/standard library ever; because "what about existing tools that won't immediately support it?", I guess. It is unbelievably unconvincing.

There is literally a fundamental analogy here between any hypothetical new expression nodes, and any new standard constructs in general — with the most notable recent example being what happened with TimeOnly and DateOnly: They got added to the standard library, tools like EF, various serializers, etc. didn't immediately have support for them, so any usage in such unsupported contexts resulted in either a runtime exception or unexpected behavior at the time, until support gradually got added. How would new expression nodes be fundamentally ANY different than that?

Notice that according to the same logic as the one that we're being provided here as to why expressions are stuck at the C# of 10 years ago, apparently the addition of TimeOnly and DateOnly would've been a heinous, unthinkably nightmarish idea, because:

The significant cost of implementing [TimeOnly and DateOnly] does not come from the [standard library]. It's fairly easy to do in the [standard library]. Its significant cost comes from every downstream tool now being broken because there is a new [built-in type] in the [framework's standard library] they were not expecting, and us needing to work with EF and other consumers of [user code] to create a way to signal [framework] version support.

It gets even worse, this argument is, in fact, even LESS applicable in the case of expression trees, because all consumers of expression trees (most prominently EF), ALREADY ONLY support a subset of "all possible expression trees". An expression tree consumer not supporting a certain thing at a given time is already the status quo, and as such, fully expected.

Funny. Some people in this thread (e.g. @olmobrutall here) tried to point out the same thing, heck, even some comments on related Stack Overflow questions made this point, and made more basic logical sense than anyone from the other side here. I'm just trying to help reveal the folly in this reasoning more clearly and bring more awareness to the issue.

[CyrusNajmabadi]

Could the entirety of expression trees be replaced with a mechanism that would produce Roslyn SyntaxNodes?

No. SyntaxNodes are just raw syntax. They do not contain important, necessary, semantics (like "what on earth does this foo syntax mean here").

[AartBluestoke]

Why doesn't the Roslyn compiler itself suffer from the same difficulty? Every time a new SyntaxNode type is added, isn't there some potential to break the shape of the parsed tree, and thus break any parsing code? How are language features ever added to C#?

But i think it does - new code structures can't be compiled using old roslyn's, they would return a syntax error.
The new bytecode output is fully backwards compatible with the runtime - and that is where massive effort in Roslyn is, and outcomes of executing that compiled code don't change (up to performance, and a few documented "undefined" outcome cases like what value does an uninitialized value hold)
I think the problem here is that new AST's have to be run through old AST interpreters.

The compiler is also not constrained to emit the same byte code from version to version, only that executing the bytecode comes with the same outcome (eg, i think how errors are handled in async can produce different bytecode in different roslyn version, but no effective difference in outcomes other that performance considerations)

An old AST might have a switch (node N){
typeof(this): x=this.thing
typeof(that): x=that.foo
... covering examples over all known examples
}
and no null check on x here. a new node type causes acrash.
even any existing code being interpreted as the new node type would cause a crash (eg, upstream library introduces an optimised 'null_coalesce ' node, which then gets passed to the runtime.

In this scenario, the source code hasn't changed, and the destination framework hasn't changed, just a library somewhere in the middle of a stack getting a version bump which "only" contains optimisations could be enough to induce runtime AST crashes.

There is also the development cost - issues with Roslyn interpretation are caught with red squigllies as you type. AST's are pure interpretation, so you only find out you've done a 'silly' at runtime, potentially quite a bit later in the dev cycle.

[ranma42]

Analyzers and/or compile-time translation of the expressions might help detecting what unsupported expressions.
These two are new-ish and still very much in progress in several contexts (see, for example, the ongoing work to translate queries AoT in EFCore), but maybe they could help in avoiding the issue of runtime failures.

[olmobrutall]

The code won’t fail if you don’t use the feature.

How do you communicate "the feature" to customers? Basically how can an API, documentation, etc ... properly communicate what type of C# syntax is / isn't legal. (…) Basically it's not about APIs supporting specific expression tree types, it's about supporting the types in the many different contexts in which they can occur.

What expression trees is a LINQ provider going to accept is a grey area already. Some mini-LINQ providers maybe only accept MemberAccessExpression because they are only meant to refer to properties (like strongly-typed reflection) while others could support almost everything. I can see that is tempting for you to be able to restrict this at compile time with some versioning or a massive attribute enumerating the supported ExpressionNodeTypes but it is very complicated, think of this query:

Query<Person>().SelectMany(p=>p.Age>18?p.Cars:p.Parents.Cars)

The ?: maybe is accepted in many context but not here because you cant use a SQL CASE for the source of a query.

Even without considering new nodes, the developer already has to know which methods and properties are supported by the LINQ provider and what not, this is a much bigger gray area anyway.

So… just not solve this communication issue. LINQ to expressions base some pitfalls
but is still very useful.

This is further complicated because there is no specification for how expression tree nodes are generated. The original native compiler implementation was effectively a 1:1 mapping of internal data structures to nodes. Consumers are incredibly fragile to the shape of the generated trees (basically fragile to internal data structures of the native compiler).

Roslyn nodes are full fidelity, so I can see there is tension a versioning issues.

Expression trees on the other side are more abstract and easily consumable:

• there is no trivia
• no difference between whether you call an extension method as static or as instance method
• no difference between query syntax or method chasing

Also, the API validates at run-time the types of the arguments etc quite strictly, so there is not a lot of freedom to make another valid expression tree really. I can only think maybe inserting/removing some unnecessary castings or a different method overloads resolution (already happening?)

Could you make an example of c# code that could produce two different valid expression trees and would benefit from a change?

The best way we've found to model this is treating expression trees like language versions. Essentially you can pick the language version that expression trees are generated with and consumers can declare what is the max version, or set of versions, they support. That version would end up getting passed along with the tree so that consumers could better handle the nodes. That gives us better tools to push consumers and produces to a place of goodness, not guessing.

The original spirit of the expression tree api was to implement like 40% of the language (only expressions) to provide 90% of the value when translating to SQL, and I think this was very wise.

Maybe there are some use cases for a new SuperExpressionTrees that supports statements and all the new features (shaders?, auto-derivation for tensors computations?) but the change will be massive and not backwards compatible, so if it ever gets done should be an different API and not an expansion of the current one.

But if LINQ was designed with the same spirit but today I think they would have increased the amount of language supported just a little bit more:

• Support ?. navigating any reference in SQL using . produces a LEFT OUTER JOIN. If the references is nullable this is semantically a ?.

Query<Person>().Select(a=>a.Car?.RegistrationDate)

but this is not supported.. so you have to write !. instead… semantically crazy but works most of the time… except in the select statement

Query<Person>().Select(a=>a.Car!.RegistrationDate) // works… except of the Car is null, then you get a cast exception from null to DateTime

some other additions that don’t even require new exprsssion nodes:

• Support short syntax for tuples.
• Support optional parameters.

This is why it is not just like adding DateOnly or TimeOnly. Expression trees aren't measured in that simple way. The nodes exist in a fragile ecosystem with heavily intertwined decisions.

I need an example to see why is this so complicated, the current compiler is able to translate Roslyn nodes into old expression trees. I don’t think minimally increasing the target nodes would be so hard but an example would help.

If you use the feature, and is implemented in the LINQ provider is all 🦄 and 🌈

Sure but how do you discover what the provider supports? Type some code, execute, get an exception, change the style a bit and repeat until success? That doesn't sound like a good developer experience.

As I mention, knowing what is supported is very complicated and the list of supported members is more important than the list of nodes. A Roslyn analyzer is probably a much better solution than versioning at the compiler level… but is very challenging because the Queryable API is shared.

If you all can accept the idea of implementing a small increase of the current expression api, instead of a big change for a more ambitious but risky change I’m willing to prepare a PR. I made it already in C# 5 or 6 back in codeplex times

[WhitWaldo]

It's been a minute since I looked at it, but there's already a fork of Roslyn that's been outfitted to support all the language changes up through C# 11 from Bart de Smet here.

I'm going to have to read through it again and compare to the various points of feedback above as it doesn't use a version number and otherwise works around the System.Linq limitations.

[colejohnson66]

Breaking out a tangent, but: Is there a reason a "new" expression tree system couldn't be "just" a 1:1 mapping of the AST itself? Why do they need to be representative of the "lowered" code? I hate to bring Rust into this, but the input to their "proc macros" are nothing more than a (brace-balanced) token stream. It doesn't even need to be legal Rust syntax.

I know expression trees aren't designed for this, but they're almost perfect for creating DSLs — you can write an interpreter around the nodes to parse out logic. That's basically what EF consumers are doing — turning a C#-like DSL into SQL. And, as C# continues to evolve, this C#-like DSL continues to stray further and further away. If expression trees were merely the AST without any "lowering", you could write much more powerful DSLs, and you wouldn't need to be worried about how compiler optimizations affect the output tree — because the AST comes before optimization.

Basically, rip off the band-aid and create something new. Such a new system would obviously need to be documented as non-exhaustive, and developers should be encouraged to document what their inputs must be. But by being the AST, and not the lowered/optimized code, new compiler optimizations won't affect inputs.

In the future — perhaps — macros could be explored as, essentially, Roslyn preprocessors, using this AST.

[erik-kallen]

This could be done with source generators and interceptors, I assume.

[colejohnson66]

@CyrusNajmabadi How would it expose your internal model? I’m suggesting exposing the raw tokens, not the semantic model. There’s no such thing as “member access”, just a period token. Perhaps, initially, the compiler could ensure it’s valid syntax. As @erik-kallen said, EF consumers could either consume them at runtime or use source generators to generate code at compile time. That would easily lend itself to macros using source generators (where the compiler wouldn’t enforce syntax validity).

[CyrusNajmabadi]

If it's the raw tokens then it's the concrete syntax model, not the abstract one.

[colejohnson66]

And why shouldn't that be exposed? Other languages manage just fine with exposing token streams.

[CyrusNajmabadi]

And why shouldn't that be exposed?

1. Because no one is asking for that. Expression tree consumers, for example, make use of hte semantics associated with the nodes.
2. Because it would be exposing a C# token model as 'the model' everyone would code to. Expression trees work regardless of language. They are lang agnostic.

You are fundamentally asking for somethign else now. And we don't expose something as major as that just off of one person asking for it, with exactly zero consumers saying it's what they actually need over expr trees.

[EmperorArthur]

Hello, just wanting to make sure I'm on the same page, since there are seemingly two different discussions going on. Forgive me if this is re-hashing the old points again, but I feel this might help more than me.

There are fundamentally two different parts to this discussion.

Things Currently Supported By Raw Expression Nodes, But Not Supported In Lambda Expressions

This includes things like throw, which map 1:1 with existing nodes. It also includes things which might be more efficient with new nodes, but could be converted to existing nodes. Like the null-coalescing operator (??) and null-conditional operators (?., ?[]).

CSharpExpressions was already mentioned as an example of solving some of those by implementing new nodes, but allowing most nodes to be reduced to a more complex tree using existing expression nodes. However, another example which does get used in production is LINQKit. Because it's the only way I can easily call a typed lambda expression from another lambda expression.

Things Not Currently Supported By Raw Expression Nodes

These are things which would explicitly require new node types to work. An example noted by CSharpExpressions is below.

Also note that some language constructs such as indexer initializers introduced in C# 6.0 can't be provided as reducible extensions to the LINQ APIs

My Issue / Question

Someone please correct me if I am misunderstanding, but I don't understand the controversy of fixing the first issue. The only possible downside is it would allow someone to more easily write expressions which contain some node types not supported by all query transformers. However, it is possible today to write those same not supported expressions today. They just have to be done by hand.

Meanwhile, the convenience of the null-coalescing operator (??) and null-conditional operators (?., ?[]) can't be understated. When given the choice between materializing a SQL query so all language features can be used, or writing 15 lines of ternary expressions most programmers will choose the former. Yet this can easily lead to horrendous access patterns.

Plus, without LINQKit everything either has to be in one large lambda, violating almost every code readability standard out there, or everything which calls something else must be a hand constructed expression. Which is something most C# programmers, including the rest of my team, aren't familiar with and can't work on.

Would someone please explain to me why fixing this first issue is so controversial.

[marchy]

You should just rip off the bandaid and start pushing updates to the API.

If you want to play it safe - do it on a small surface area of new language features to give the community / providers time to update for compatibility.

Then on the next C# release push out the next wave of updates once library authors get the gist of what’s to come.

Stop overprotecting the community though. We will pick up the pieces. PR’s will be raised, compatibility issues will be fixed. In MUCH less time than it takes you guys to debate all the downsides of how to minimized breakage.

Let’s get on with it already!

[Atulin]

Last I heard, backwards compatibility will remain a priority because big players are paying big money for support, and breaking their .NET Framework 3.5 projects would turn big money into slightly-less-big money as they move somewhere else.

Us, little ones, we just enjoy the crumbs off the table.

[colejohnson66]

Anyone on .NET Framework is not affected by changes in .NET Core 1-3 or .NET 5+