Should we change PEP 484 to disable implicit Optional when default = None?

[gvanrossum]

CC: @ddfisher @JukkaL @markshannon

PEP 484 currently says:

An optional type is also automatically assumed when the default value is
None, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None) -> None: ...

This was intended as saving some typing in a common case, but I've received strong feedback from some quarters that this is not consistent and a bad idea. There are other places where None is allowed but none of them automatically add Optional (to the contrary).

So far it hasn't mattered much for mypy users because mypy doesn't have Optional support, but that's soon going to change (the --strict-optional flag is becoming more reliable) so if we're going to change this, now would be a good time. Thoughts? If we don't change this soon it will probably be too late.

[ddfisher]

I'm for this change. If I recall correctly, this is the only place where the PEP suggests typecheckers understand types to be something other than what's written as the type, so I think this change would be a nice simplification.

I've also started to notice this cause a bit of user confusion. See python/typeshed#500, for example, for all the arguments to builtin functions that were accidentally marked Optional due to this.

[ilevkivskyi]

I have seen some confusions between optional arguments and optional types, and the convention in question amplifies such confusions. Therefore I am in favour of removing it.

[gvanrossum]

OK, let's bring this up on python-dev once Python 3.6 beta 1 is released
(i.e. after next weekend).

On Sat, Sep 3, 2016 at 7:32 AM, Ivan Levkivskyi notifications@github.com
wrote:

I have seen some confusions between optional arguments and optional types,
and the convention in question amplifies such confusions. Therefore I am in
favour of removing it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#275 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMpUgUvnteOgZfJvI0g_D50N3TOthks5qmYTugaJpZM4J0K87
.

--Guido van Rossum (python.org/~guido)

[dmoisset]

IMO most of the confusion comes from the name being wrong. An argument is «optional» when the parameter has a default in python. An «optional» type is a type that you can choose to add or not (as in gradual typing). If we had Nullable[T] instead of Optional many of these confusions wold go away (and it's the name used mostly everywhere else, other than Maybe T in Haskell).

Other than that, there is an inconsistency that may require a change (I just wanted to clarify that the confusions shouldn't be part of the argument)

[prechelt]

@dmoisset: I agree that Nullable or Nonable (1 char shorter) or Maybe (3 chars shorter) would be better solutions than Optional.

On the other hand, even the short Maybe is cumbersome. Nonable parameters are a frequent case and I believe the =None is sufficiently explicit to warrant introducing an additional PEP rule to cope with it, even if that rule has the character of an exception.

[markshannon]

I like that the Optional[] can be omitted if a None default is provided. It is less typing, reads naturally (to me) and is unambiguous.
If users want to use Optional they can since Optional[Optional[T]] == Optional[T].
As syntactic sugar goes, it seems quite sweet :)

OOI, which quarters?

[JukkaL]

I haven't had enough experience with using mypy strict optional checking to decide whether the extra verbosity would become awkward. Maybe we should wait until we have a substantial codebase that can be type checked using strict optional checking.

[markshannon]

Why do you use the word "strict"? That suggests that there is something "loose" about the syntactic sugar. There isn't; it is precisely defined.

[JukkaL]

Strict optional checking means that mypy treats None as a regular type, and mypy enforces that you add guards like if x: foo(x + 1) for values with an Optional[] type. Without strict optional checking, None is implicitly compatible with every type, kind of like null in Java, and an Optional[X] type is equivalent to just X. It has nothing to do with syntax.

[markshannon]

So how is it relevant to default values for parameters making the type Optional?

[JukkaL]

The reason for adopting the syntactic sugar was the worry that otherwise type annotations would be too verbose and thus maybe also harder to read. I think that the main arguments for not having the syntactic sugar are consistency and conceptual simplicity.

If we have a codebase that uses Optional[] consistently, which can be enforced by the strict optional mode of mypy, we could evaluate, at least for that codebase, whether the arguments hold water. We can get a realistic of estimate of the actual verbosity of annotating code with or without the syntactic sugar. We can perhaps also get a better understanding of whether the inconsistency is an actual problem for people who are working on the code.

It may all boil down to personal preference, but having some empirical data could be useful for the discussion. Also, it doesn't seem to me that this is an urgent thing to resolve, as this doesn't require major typing implementation changes (though get_type_hints may be affected) and it's going to be easy for a static checker to support both modes via a configuration option. I don't see a need to rush the decision, and we can first wait for tools that can perform useful "strict optional" checking to be built.

[markshannon]

Whether it is easier to read is, as you say, personal preference, which is rarely influenced by empirical evidence 😄

Happy to wait and see what you come up with.

[ilevkivskyi]

@JukkaL I ma not sure what kinds of inconsistency you mean, but this one

x = None # type: int # here the type stays int, unlike for function params

will go away after PEP 526. I think there it is better to assume implicit Optional or not for both of these

def f(x: int = None): ...
x: int = None

but not separately for each of these.

[markshannon]

x = None # type: int

Ah, that thing. I think that should never have been added. It is just self-contradictory.

[JukkaL]

@ilevkivskyi There's still the question about annotating instance variables with mutable types in the class body in pre-3.6 code. This is what mypy users can write today:

class A:
    x = None  # type: List[int]  # Can't use x = [] (list is mutable) and don't want Optional

    def __init__(self) -> None:
        self.x = []

This is a common use case and I think we need to have a reasonable way to expressing this without PEP 526 syntax.

[markshannon]

That seems somewhat off topic, want to open a new issue?

[JukkaL]

I'm not proposing to change that, I just brought it up as I think that it's related to this consistency discussion and one of the more common cases where things currently feel a bit clumsy. PEP 526 would resolve that inconsistency for new code that can assume Python 3.6+.

[ambv]

Internal feedback at Facebook also suggests we should make this change.

[ddfisher]

There's been some internal feedback at Dropbox to that effect as well.

I think if the status quo was the other way, it's very unlikely that we'd want to add this syntactic sugar. I think the main reason we still have it is due to status quo bias. This change should still be low cost to make -- let's make it now, before that changes.

[gvanrossum]

I agree we should not have done this. I'm more worried about the cost of updating 600K LoC internal code (plus numerous stubs) than @ddfisher. It's probably a simple PR so we can try it out really cheaply.

[ddfisher]

On our internal repo that uses None Checking more, a run with this change causes about 20 errors in Python 2 stubs (13 of which are in builtins) and 120 errors in the rest of the codebase. Thankfully, these errors are all fixable mechanistically -- I'd be fine with making the fixes to our internal repos.

[gvanrossum]

What should be the next step? Maybe a simple thing would be to add a flag to mypy that disables the default Optional. That will allow us to track down all the stubs that rely on this; once we have those fixed (and our internal repo) we can make it the default.

At the same time we should propose a diff to PEP 484 and discuss it. At this point I expect that would be pretty uncontroversial.

[Tinche]

Coming here via Guido's Python-Dev post.

I'm a strong +1 on explicit Optionals. I use them heavily in our internal codebase and I consider them very helpful. They are verbose though, I'd be totally on board if we could get a shorter name for them and deprecate Optional.

[markshannon]

To clarify, the proposal is that this becomes a type error:
def handle_employee(e: Employee = None): ...
Is that correct?

[ilevkivskyi]

To clarify, the proposal is that this becomes a type error:
def handle_employee(e: Employee = None): ...
Is that correct?

Yes.

[JukkaL]

I also prefer requiring explicit Optional[...] types, and I'd like a shorter syntax. However, the shorter syntax is a separate issue.

[JelleZijlstra]

We fixed all instances of implicit Optional in typeshed a while ago.

@gvanrossum is the next step here just to change PEP 484?

[gvanrossum]

We're nearly there, I think we have a few steps to go first:

• Add --no-strict-optional somewhere so that typeshed stays clean
• Add words to PEP 484 allowing (maybe even encourage) type checkers to do this
• Discuss on python-dev when type checkers should do this

[JelleZijlstra]

The first one is done: https://github.com/python/typeshed/blob/master/tests/mypy_test.py#L134.

I submitted a patch to PEP 484 in python/peps#689.

[ilevkivskyi]

I assume we will keep a flag in mypy, but will just invert the default. Right?

[gvanrossum]

I assume we will keep a flag in mypy, but will just invert the default. Right?

Yes we should do that.

[gvanrossum]

Closed by python/peps#689.

[sproshev]

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

Should I use some another value to achieve this? If so, should it have type int or other one? E.g.:

class _MISSING:
    pass

def f(p: int = _MISSING):
    ...

[ewjoachim]

I'd say it's not desirable to put a default value if you consider that default value is not an acceptable value. In this case, just don't put a default value. If you don't pass anything, mypy will tell you and you'll get a type error, and if you explicitely pass None, mypy will tell you it's wrong.

def foo(p: int):
    # in the function body, don't try to defend against p being not an int
    ...

[sproshev]

What about other cases such as dataclasses.field? Default value for default and default_factory has incompatible type. See stub and specification.

[keu]

@ewjoachim there are these different scenarios:

# None is not one of possible values, func1 can't be called with explicit `param = None`
def func1(param: Class = None):
   param = param or Class("somethin")

# None is one of the possible values, funct2 can be called with explicit `param = None`
def func2(param: Optional[Class] = None):
   pass

# None is one of the possible values, funct3 requires param and can be called with explicit `param = None`
def func3(param: Optional[Class]):
   pass

# None is one of the possible values, funct3 can be called with `param = None`
def func4(param: Optional[Class] = obj):
   pass

i.e. default values has nothing to do with type annotation i.e. public contract of a function

[ewjoachim]

Not sure I get your point, and I don't agree with "default values has nothing to do with type annotation":

This function has wrong annotations which we can tell BECAUSE the annotations and the default value are clashing:

def func(param: int = "hello"):
   pass

---

I was answering to:

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

and I understood "does not accept None" as "if you call the function with None (or without an argument), the call is going to be rejected. And I stand by my (... 2-year-old-)opinion: if a function is written to fail if an argument is missing, it's best to not have that argument have a default value.

If you want a default value and you want to be able to detect that the default value was used, either use : Optional[...] = None if None doesn't already have another meaning or do something like:

missing = object()
def func(a: Any = missing):
	if a is missing:
		print("default")

or

class Missing:
	pass
missing = Missing()
def func(json: Union[None, list, dict, str, int, float, Missing] = missing):
	if json is missing:
		print("default")

Maybe I missed an important point ?