Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Champion "Nullable reference types" (16.3, Core 3) #36

Open
2 of 5 tasks
MadsTorgersen opened this issue Feb 9, 2017 · 259 comments
Open
2 of 5 tasks

Champion "Nullable reference types" (16.3, Core 3) #36

MadsTorgersen opened this issue Feb 9, 2017 · 259 comments
Assignees
Labels
Implemented Needs ECMA Spec This feature has been implemented in C#, but still needs to be merged into the ECMA specification Proposal champion
Milestone

Comments

@MadsTorgersen
Copy link
Contributor

MadsTorgersen commented Feb 9, 2017

Note this includes the object constraint: where T : object.

LDM:

@gulshan
Copy link

gulshan commented Feb 12, 2017

Will default constructor also generate warnings for all non-initialized reference type members?

@iam3yal
Copy link
Contributor

iam3yal commented Feb 13, 2017

In the following case:

var y = b ? "Hello" : null; // string? or error
var z = b ? 7 : null; // Error today, could be int?

I think that in both cases it should be <type>?; more and more people are using var in their codebase and switching to use a type just to evaluate an expression seems odd, however, in the case you really want it to be an error, using a type actually make more sense.

@gafter gafter added this to the 8.0 candidate milestone Feb 22, 2017
@yaakov-h
Copy link
Member

Nullability adornments should be represented in metadata as attributes. This means that downlevel compilers will ignore them.

Can we have a serious discussion about using modopts (or even modreqs) to allow for nullability overloads, i.e. two methods who only differ by nullability?

This would be useful for functions where null input = null output, and non-null input = non-null output, or even vice-versa.

@HaloFour
Copy link
Contributor

@yaakov-h

I believe that conversation has happened a couple of times already. modreq is not-CLS. modopt does allow for overloads but requires specific understanding on the part of all consuming compilers since the modifier must be at the very least copied into the call signature. Both break compatibility with existing method signatures. For something that will hopefully proliferate across the entire BCL very quickly using modopt would create a big hurdle to doing that.

@gulshan
Copy link

gulshan commented Mar 9, 2017

I think this feature is about objects and not types as it does not change anything about the actual type system. When nullability was introduced for value types, it was about the types. But that's not the case here. Quoting the proposal (emphasis mine)-

This feature is intended to:

  • Allow developers to express whether a variable, parameter or result of a reference type is intended to be null or not.
  • Provide optional warnings when such variables, parameters and results are not used according to their intent.

I think the "variables, parameters and results" (are the members of a class being excluded here?) can be easily replaced by "objects". The proposal is introducing non-nullable objects(even including value types) and putting them to be default with separate notation for nullable objects. So, the proposal can be renamed to-
"Default non-nullable objects" or something like that IMHO.

@gafter
Copy link
Member

gafter commented Mar 9, 2017

@gulshan Objects exist at runtime. Types exist at compile-time. This is about changing what happens at compile-time, not at runtime.

@gulshan
Copy link

gulshan commented Mar 10, 2017

@gafter I have two questions-

  • Would typeof return same type for nullable and non-nullable references?
  • And, as default returns null for classes, will FooClass foo = default(FooClass) generate warnings?

For nullable value types, both the answers are "no". But, I guess for nullable references, the answer is "yes". Because there is nothing changed about types. Only thing changed is, whether a reference should hold null or not. If I am wrong here, please correct me.

BTW, now I propose the title "Default to non-nullable references" or simply "Non-nullable references".

@gafter
Copy link
Member

gafter commented Mar 10, 2017

@gulshan Yes, and yes. The first because System.Type objects only exist at runtime. The second because the compiler knows that default(SomeReferenceType) is the constant null.

@Thaina
Copy link

Thaina commented Mar 14, 2017

Forgot to ask that. Is this feature would allow generic nullable type?

I mean code like this should be able to compile with this feature enabled?

public T? GET<T>(string obj)
{
    try { return HttpGet(url).Convert<T>(); }
    catch { return null; }
}

int n = GET<int>(url0) ?? -1;
var m = GET<MyCustomClass>(url1);

@gafter
Copy link
Member

gafter commented Mar 14, 2017

@Thaina Not as currently envisioned, no. There is no way to translate that into something expressible in the CLR (except by duplicating the method for each "nullable" unconstrained type parameter).

@orthoxerox
Copy link

orthoxerox commented Mar 14, 2017

@gafter how would duplicating work if C# can't support methods that have mutually exclusive constraint flags (gpReferenceTypeConstraint and gpNotNullableValueTypeConstraint) but have otherwise identical signatures from the viewpoint of overload resolution?

@yaakov-h
Copy link
Member

yaakov-h commented Mar 15, 2017

The proposal doesn't take into considerations any of the discussion from the roslyn repo about telling the compiler "I know at this point in time that the value is not null. Do not warn me.", for example using a ! postfix operator.

Can we take that into consideration, or will #pragma warning disable be the only way around false positives?

@Richiban
Copy link

@gafter In @Thaina 's example, would it be appropriate to lift the return into a Nullable type at that point? I'm not sure of the feasibility of a compiler feature that sometimes works with Nullable<T> and sometimes works with plain T and hides the difference between the two to the compiler.

I was thinking that the original example:

	public T? GET<T>(string obj)
	{
	    try { return HttpGet(url).Convert<T>(); }
	    catch { return null; }
	}

would like this if you decompiled it:

	public Nullable<T> GET<T>(string obj)
	{
	    try { return HttpGet(url).Convert<T>(); }
	    catch { return null; }
	}

But in C# vNext you wouldn't know. The type just appears to be T? still.

@MikeyBurkman
Copy link

I'm still a bit curious how this could possibly be implemented with var without possibly breaking a lot of existing code:

var s = "abc";
...
s = null;

This is perfectly valid currently. However, what type should s be inferred to? If it's inferred to just String, then it'll break existing valid code. If it's inferred to String?, then it'll essentially force devs to make a choice between type inference and strict nullability checks.

@Pzixel
Copy link

Pzixel commented Mar 27, 2017

@MikeyBurkman it's may be inferred in different ways based on project checkbox (like it's done for unsafe)

@mlidbom
Copy link

mlidbom commented Mar 27, 2017

To my mind the most important issue, by far, to fix is the current need to manually validate that every single reference type parameter is not null at runtime. This latest proposal seems to be de-emphasize that to the point where it is just mentioned as a possible "tweak" at the end. The way I understand it you would get warnings but zero guarantees.

I believe I am far from alone in feeling that without fixing that this feature would be of limited value. Here is a vote for ensuring that runtime not-null guarantees are prioritized. It would probably remove something like 90% of the current parameter validation code that that C# developers write.

I would suggest going a step further than the "!" syntax and supplying a compiler switch that would automatically generate such checks for ALL reference type parameters not explicitly allowed to be null by marking them with "?". This would be a huge time saver and if implemented in the compiler it should be possible to implement optimizations that would make the cost negligible.

I would also suggest extending the "!" syntax to check for default(TValue) in struct parameters. So that you could, as one example, validate that a Guid is not Guid.Empty with just a single "!" character.

@mattwar
Copy link
Contributor

mattwar commented Mar 27, 2017

@MikeyBurkman the var can infer the type to be string? with a known non-null value. This means the variable will pass all the tests and be able to be used as not-null until it is changed to the null value or a value not known to be not null.

@mattwar
Copy link
Contributor

mattwar commented Mar 27, 2017

@Richiban you could only write that if T was either constrained to class or struct.

@MikeyBurkman
Copy link

@mattwar The issue with inferring it to the nullable value isn't that it's necessarily wrong, it's that it's extremely inconvenient, to the point that it becomes a nuisance. I don't recall seeing any suggestion about flow typing (where a variable's type can be narrowed by, for instance, a non-null assertion), so in order to pass that variable to a function not expecting a null, I'd either have to assign it some default value (foo(s || "")) which would be quite redundant in most cases, or I'd have to just assign it the non-null type explicitly, making var useless.

@HaloFour
Copy link
Contributor

@MikeyBurkman

There have been multiple conversations on the Roslyn forums about this subject. The "nullability" isn't part of the type of the local, it's how it's used. So the following would produce zero warnings:

string? s = "1234";
int len = s.Length; // the compiler knows that s is not null here

So in most circumstances you wouldn't be forced to do anything special with the variable in order to placate the compiler, as long as the variable was definitely assigned to something that wasn't null.

@Pzixel
Copy link

Pzixel commented Mar 27, 2017

@mlidbom I'm against the ! syntax becuase it provides a lot of mess. Syntax should be consistent thus we need ? for nullables and just a type for common variables, but not ?/type for value types and type/! for reference ones. It's better to have a little breaking change than open this bottle with djinns. All constants must be of not-null types (it's obvious, because otherwise you just cannot have non-null constants), thus var x = some_constant should be of non-null type even if in previous language version it is inferred as nullable.

Another question if non-null type will be a csc part without CLR support or we will have it? I mean readonly is a csc hack and reflection allows to modify readonly field. Are we going to do the same with non-null references when you cannot pass nulls in C# but fully able to do it from reflection?

@ceztko
Copy link

ceztko commented Oct 27, 2018

Its very similar to saying, for example, that C++20 will block multiple inheritance and spam warnings all over the place if you use a -zaelotclass compiler switch, simply because a bunch of people in charge of C++ think that multiple inheritance is heresy and not a proper thing to do.

@hades-incarnate I think your example is not correct, as it would have the same impact of spamming warnings just by using goto statement because some one thinks is wrong by principle. Nobody is trying to criminalize the use of nullable types. As @CyrusNajmabadi pointed out very well, the most common use must be favored over the less common one. If you still aren't convinced (and apparently I'm the first to cite here) try to read what Tony Hoare, Turing Award winner and father of Quicksort algorithm, have to say about null references (same wikipedia article):

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

So, the same language design choice you defend asserting it is worth billion dollars of investments in learning and preserving, for one of the biggest geniuses in computer science it actually caused billion of dollars in damage.

Sometime is useful to remember we are just dwarfs standing on the shoulders of giants.

@theunrepentantgeek
Copy link

Tony Hoare understates the cost just a little.

I saw an article in CACM some years ago that estimated the economic cost of null pointer bugs as being in excess of a $US billion per year in the United States alone.

This is why I'm personally very excited to see nullable reference types coming soon.

(Apologies: I had a quick look to see if I could find the article to link, but wasn't able to do so.)

@limonana
Copy link

limonana commented Nov 9, 2018

I read the proposal and a few of the LDMs and I still don't understand how can I gradually adopt the feature without performing any migration?

let's say I am upgrading my solution to C#8, I am using treat warnings as errors and I want to set the feature flag on, what I need to do in order for solu to compile?

What I understand is if I turn the feature flag on,oblivious reference will become non-null references. am I correct? if so what happens with existing libraries and the .Net framework.?

Where the feature flag is set? is it per solution? per project?

@svick
Copy link
Contributor

svick commented Nov 9, 2018

@limonana

what I need to do in order for the solution to compile?

Nothing, this feature doesn't produce errors, only warnings.

oblivious reference will become non-null references. am I correct? if so what happens with existing libraries and the .Net framework?

Oblivious references will become non-null references in your code. If you use a non-annotated library, that will stay oblivious, I believe.

Where the feature flag is set? is it per solution? per project?

There will definitely be a per-project flag and also some way to set it on a more gradual basis. I think it's not clear yet how exactly that will look, these LDM notes should contain the latest information.

@gafter
Copy link
Member

gafter commented Nov 9, 2018

There are a number of ways of easing a large code base into this feature. Here is one sequence of steps:

  1. First, turn on (enable) the warnings associated with the feature, but without enabling unannotated types meaning non-null. Do not turn on the "W" warnings, which are ones that would complain about whether or not local variables are "properly" annotated. The "W" warnings do not help catch problematic uses of null values, so I don't think they are that valuable, and in any case they interfere with migration. Then rebuild. At this stage you'd get warnings for problematic use of external APIs that are annotated (e.g. passing null to a non-null parameter, or dotting through a result that was declared to be possibly null), and you'd also get warnings from code that the compiler can detect is locally incorrect (e.g. assigning a variable to null and then later dotting off the variable).
  2. Next, start opting in specific APIs (internal and/or external) in your code, by using #nonnull safe before and #nonnull restore afterwards. Be sure to add ? for reference types in signatures that might be null, and leave it off where a reference type ought not to contain a null value. Rebuild, and notice any new warnings from uses of these APIs, and adjust callers to remove any bugs found.
  3. Repeat step 2 until the whole project is migrated.
  4. [Optional] Turn on "W" warnings for the full project, and use the compiler warnings to adjust the declaration of local variables so that the type of those that might contain null are properly annotated. This step might be a substantial amount of work, but is unlikely to expose additional misuses of null that would result in runtime failure.

@YairHalberstadt
Copy link
Contributor

What is the default value for a struct containing non-nullable reference types? Since all structs are zero-initialized will they be null?

@PathogenDavid
Copy link

The fields would be null, that doesn't change. Accessing the non-null fields should result in a warning once roslyn#30731 is fixed.

It seems to me that returning a partially-initialized default struct or calling methods on it should result in a warning, but I don't see any issues relating to that.

@endofunk
Copy link

endofunk commented Mar 3, 2019

I think it's a pity that the implementation of nullable reference types doesn't appear to permit extension methods, as is possible with value type's Nullable<T>.

In contrast Kotlin, Rust and Swift have implemented value and reference nullable types in a similar fashion to C# Nullable<T> for value types; meaning it's fairly easy to add extension methods like:

  • Map (Linq Select)
  • Apply (applicative functor)
  • LiftA (applicative lifting operation)
  • FlatMap (Linq SelectMany)
  • LiftM (monadic lifting operation), with similarities to Linq's SelectMany that has this extension method signature, for example: public static Nullable<R> SelectMany<A, B, R>(this Nullable<A> @this, Func<A, Nullable<B>> fn, Func<A, B, R> select)
  • etc.

Workaround
The only obvious workaround is to fall back on a data type like Maybe, and avoid using both Nullable<T> and nullable references.

Is there any possibility that nullable reference types in future would support extension methods?

@CyrusNajmabadi
Copy link
Member

@endofunk

I think it's a pity that the implementation of nullable reference types doesn't appear to permit extension

What do you mean by that? For example, let's look at Swift's map. Can we not write an extension like so:

public static U? Map<T, U>(T? val, Func<T, U> func)
    where T : class
    where U : class
{
     return val == null ? null : func(val);
}

You can now have:

string? s1 = "hello";
string? s2 = null;
string? s3 = s1.map(v => v + " world");
string? s4 = s2.map(v => v + " world");

In this case, s3 should end up as hello world and s4 will be null.

@endofunk
Copy link

endofunk commented Mar 3, 2019

@endofunk
What do you mean by that? For example, let's look at Swift's map. Can we not write an extension like so:

public static U? Map<T, U>(T? val, Func<T, U> func)
    where T : class
    where U : class
{
     return val == null ? null : func(val);
}

Quite right; I just tested that successfully a few minutes ago, and it worked as expected.
Was planning to delete my comment, but thanks for the reply.

@CyrusNajmabadi
Copy link
Member

Sure!

@Joe4evr
Copy link
Contributor

Joe4evr commented Mar 11, 2019

From recent design notes:

  • Object initializer pattern is a persistent problem and the warnings don't seem to be providing much value

I have an idea that could mitigate this pain point. Let's use the (in)famous Person class to illustrate:

public class Person
{
    public string FirstName { get; set; }
    public string? MiddleName { get; set; }
    public string LastName { get; set; }
}

Right now, enabling nullable will add two warnings that non-nullable properties FirstName and LastName are uninitialized. One might then want to put #nullable disable in front, but that will produce a warning for the declaration of MiddleName: The annotation for nullable reference types should only be used in code within a '#nullable' context. Moreover, code at use-sites won't get warned for potential null-dereferencing on MiddleName.

But as the developer, you know that this type is only a DTO to negotiate between your code and a serialization framework, which the vast majority of the time will construct the object and fill in the properties via reflection prior to handing it back to you.

My proposed solution is another attribute: [DTOType] (name TBD):

[DTOType]
public class Person

This attribute informs nullable flow analysis to supress the warnings about the un-initialized properties at the class definition, but when the properties are called from elsewhere, the nullable annotations are respected as normal.

So what about the object initializers?

The slightly rarer cases that you call new Person in code explicitly, it's pretty much always followed by an object initializer to put values in. So in order to facilitate OI remaining useful syntax candy, the same attribute is used to inform callers that they have not set a value to a non-nullable property prior to passing it around:

var p = new Person
{
    FirstName = "Joe"
}; //warning: Non-nullable property 'LastName' remains uninitialized.

In addition, if during development it becomes apparent an additional non-nullable property has to be added to Person, all the constructing call-sites are notified that a value needs to be given.

Thoughts?

@YairHalberstadt
Copy link
Contributor

@Joe4evr
Sounds good.
Two thoughts:

  1. Do we want to restrict this to DTOs? Maybe have an attribute on an auto property/field which indicates the field must be initialised by whoever constructs the object.

  2. I think we should use the same rule as structs. All such fields must be initialised before the object is used, or a warning results. It doesn't necessarily have to be in an object initialiser.

@GeirGrusom
Copy link

At least EFCore and Json.NET supports building a type through its constructor which means that the DTO could be made immutable which should resolve the issue. Though I understand that this isn't applicable to all serializers, and immutable DTO's can be a bit of a back pain sometimes, but with that it should be a non-issue.

I don't understand why you should be able to not assign a non-nullable field though? Just make the property nullable instead of making a pinky-swear type.

@dgreene1
Copy link

+1 to everything @GeirGrusom said and bravo on the term “pinky swear type.” I love it!

At least EFCore and Json.NET supports building a type through its constructor which means that the DTO could be made immutable which should resolve the issue. Though I understand that this isn't applicable to all serializers, and immutable DTO's can be a bit of a back pain sometimes, but with that it should be a non-issue.

I don't understand why you should be able to not assign a non-nullable field though? Just make the property nullable instead of making a pinky-swear type.

@Joe4evr
Copy link
Contributor

Joe4evr commented Mar 11, 2019

I don't understand why you should be able to not assign a non-nullable field though? Just make the property nullable instead of making a pinky-swear type.

I don't know if you've noticed, but the entire implementation of Nullable Reference Types revolves around pinky-swearing what is and is not allowed to be null. If your domain model demands a specific property to be non-null, then you never want to pass around an instance where that property is null.

Remember, this feature's mantra is "We can protect you from Murphy, not from Machiavelli."

@GeirGrusom
Copy link

But this is trying to completely violate the point of the future to begin with. It will just punch another hole simply for convenience.

@dgreene1
Copy link

dgreene1 commented Mar 11, 2019

The new attribute is an interesting proposal, but I agree with others who say “It will just punch another hole simply for convenience.”

To go into your proposal more @Joe4evr: Wouldn’t it be better to enhance the serializer to suppress the warning rather than accidentally providing a way for non-serializer code to bypass the null safety? (Note: I might be misunderstanding your proposal)

This attribute informs nullable flow analysis to supress the warnings about the un-initialized properties at the class definition, but when the properties are called from elsewhere, the nullable annotations are respected as normal.

I hear you, but I feel like the serializer should be capable of getting around this as opposed to introducing an unnecessary additional annotation/attribute. That’s at least my hope since most of the serialization scenarios already involve translating between a dynamic world (HTTP, JSON, or a file) into a static type. So in a sense, the serializer already knows how to ignore warnings. So if we keep that in mind, all you need to do to your Person example is to add a constructor that allows for developers to set the non-nullable fields.

Side note: @Joe4evr it’s not a pinky swear type if you make something nullable. When you denote something as nullable you’re doing the opposite of making a promise— you’re telling the consumer to validate it themselves.

In regards to your comment about “the entire implementation” I think what is wonderful about the 8 beta is that it allows you to check at the boundaries. This quote from a TypeScript best practice article (where nullable reference types has existed for years) really sums up the beauty perfectly. I can’t wait for C# to get there too:

“it is better to have strict validations at the boundaries when untyped data is coerced to typed data structures, so that the rest of our code can make strong assumptions about the type guarantees and can take advantage of a static type checker.“
~ https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/

@HaloFour
Copy link
Contributor

Object initializer pattern is a persistent problem and the warnings don't seem to be providing much value

This is because the warnings are in the completely wrong place. These types rarely have ctors or initializers because they're intended to be initialized by consuming code, and that's exactly where the enforcement should lie. Warn if the consumer doesn't initialize all of the non-nullable properties/fields before they allow the DTO to escape. I grant that this may require another attribute to annotate the types that should behave this way.

// no warnings on this type
public class PersonDto {
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

static void M() {
    var dto1 = new PersonDto { FirstName = "Bill", LastName = "Microsoft" };
    M2(dto1); // no warning

    var dto2 = new PersonDto { FirstName = "Jeff" };
    dto2.LastName = "Amazon";
    M2(dto2); // no warning

    var dto3 = new PersonDto { FirstName = "Tim" };
    M2(dto3); // warning

    dto3.LastName = "Apple";
    M2(dto3); // no warning
}

These types are incredibly common and aren't going to switch to a different paradigm. The same exact problem would apply to immutable DTOs with a mutable and initializable builder. The compiler needs to conform to how people write and use these types, otherwise the feature loses its value.

@YairHalberstadt
Copy link
Contributor

I think this might be worth moving into a separate proposal.

@YairHalberstadt
Copy link
Contributor

Created a seperate proposal: #2328

@jcouv jcouv changed the title Champion "Nullable reference types" Champion "Nullable reference types" (16.3, Core 3) Jul 18, 2019
@333fred 333fred added the Implemented Needs ECMA Spec This feature has been implemented in C#, but still needs to be merged into the ECMA specification label Oct 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Implemented Needs ECMA Spec This feature has been implemented in C#, but still needs to be merged into the ECMA specification Proposal champion
Projects
None yet
Development

No branches or pull requests