Use type hints in Python 3.x to infer variable types if possible

[zooba]

Even though Python ignores type hints at runtime and they can be pretty much any expression, we should use them to infer variable types in a function if we possibly can.

See PEP 484

(Migrated from https://pytools.codeplex.com/workitem/394)

[gnychis]

Is there any mechanism in Intellisense where we could give type hints to help Intellisense give us member variables for something?

[gnychis]

For example, can PTVS type hint like Pydev does? http://www.pydev.org/manual_adv_type_hints.html

@zooba

[theonewolf]

@zooba this would be an amazing upgrade to our workflow if PTVS supported type hinting in Python.

[zooba]

@theonewolf That's great to hear. I don't suppose you'd be able to suggest how they would help you? I have a few ideas and have not yet figured out which would be most helpful:

• warn when we infer variable types that don't match the hints
• warn when the hints don't match the inferred types
• assume the hints are correct and ignore all other inferred types
• assume hints are correct outside the function but use inferred types inside the function

"All of the above" is potentially where we'll end up, but the default setting is very important for this as probably 99% of users won't even realise it can be changed. I'm very keen to find out what benefits people hope to obtain from adding type hints so we can make an informed decision here, and given we already have one of the best type inferencing engines available for Python, we need to be able to leverage both that and explicit type hints.

[zooba]

@gnychis PTVS does have special handling for isinstance calls in assert and if statements, so if you have:

x = get_some_value()
assert isinstance(x, MyType)

We will treat x as MyType beyond the assertion. You can also use if False blocks to add stub code that won't be executed but may help PTVS understand what's happening in your code:

if False:
    x = MyType()
x = get_some_value()

(This works because we retain all values in a given variable regardless of conditions or sequencing. Only isinstance will remove types from a variable in our analysis - another assignment does not remove the previous type.)

Currently, we have not invented - and have no plans to invent - our own meta-language for type hinting.

[theonewolf]

@zooba basically I want Intellisense for everything. Sometimes the type of a variable is not known.

I would want to be able to use type hints to fill in such gaps. Flying blind gets annoying 😞

All four of your bullet points make sense for me as well.

Ultimately I'd want Mypy-style static type-checking capabilities down the road.

[theonewolf]

@zooba for your suggestions to @gnychis, would it be possible you think to create a generic function decorator that performed the assert isinstance(x, MyType) for all passed in arguments?

I've been trying that today with moderate success. We can't have typed lists for example 😞.

[zooba]

@theonewolf I don't think we preserve enough information for it to be generic, unfortunately - dictionaries in particular only keep a very limited amount of precise key-value mappings based on the key value before switching to using the key's type (probably str, as opposed to "x").

The best approach right now is to assert at the start of your function (we should support isinstance(x, A) and isinstance(y, B) and ...), or use if False and assignments (which allows you a choice of whether to include the types passed in or exclude them as you wish).

[theonewolf]

Gotcha, thanks @zooba. I think I was able to get it to work for *args, but not for **kwargs, as you mention. If we could hide it in a decorator that would be amazing 😆, but I understand the current limitations.

[den-run-ai]

@zooba do you (plan to) support the types from docstrings?

[zooba]

@denfromufa No specific plans, but we could do.

We already do a limited form of this when scanning .pyd files, so it wouldn't be too hard to expand it. The problem is that there is no agreed upon format here, so we occasionally end up with complete rubbish - PEP 484 formalizes the format, which makes it preferable.

[den-run-ai]

@zooba, type hinting is good (in fact looks like borrowed from TypeScript), but who is going to write it (like DefinitelyTyped) for numpy, scipy, sympy, pandas, scikit-learn, matplotlib, etc. in Py 2/3 compatible manner?
Right now all of them use numpydoc (derived from google docstring format) with sphinx extension. So numpydoc is probably worth supporting in PTVS.

[den-run-ai]

Here is a good example:

def parse_expr(s, local_dict=None, transformations=standard_transformations,
               global_dict=None, evaluate=True):
    """Converts the string ``s`` to a SymPy expression, in ``local_dict``

    Parameters
    ==========

    s : str
        The string to parse.

    local_dict : dict, optional
        A dictionary of local variables to use when parsing.

    global_dict : dict, optional
        A dictionary of global variables. By default, this is initialized
        with ``from sympy import *``; provide this parameter to override
        this behavior (for instance, to parse ``"Q & S"``).

    transformations : tuple, optional
        A tuple of transformation functions used to modify the tokens of the
        parsed expression before evaluation. The default transformations
        convert numeric literals into their SymPy equivalents, convert
        undefined variables into SymPy symbols, and allow the use of standard
        mathematical factorial notation (e.g. ``x!``).

    evaluate : bool, optional
        When False, the order of the arguments will remain as they were in the
        string and automatic simplification that would normally occur is
        suppressed. (see examples)

[gnychis]

I'd suggest following PEP 484 for how to style type hints. I agree with @denfromufa that this functionality is extremely nice to have. Our company started out using PTVS, but switched over to PyCharm for its support of type hints.

We can now type hint like:

def save(self, filename: str, save_as: bool, overwrite: bool) -> bool:

Not just that, but now these type hints are used directly by PyCharm's linter to throw us warnings if types don't match.

PTVS has amazing intellisense support for IronPython, but after we switched to PythonNet so that we could use Python 3.x, we lost intellisense to C# in PTVS (since it does not directly support PythonNet), and the additional support of type hints just led us towards PyCharm.

Just some feedback from our experience! We'd love to see PythonNet support in PTVS and type hints. This speeds up our development.

[zooba]

Yep, this whole item was started because of PEP 484, so that's definitely on the planning table.

Processing doc strings is fine if you require a specific format, but we shouldn't be the ones to choose that winner. We've already got some heuristics that work reasonably well for many of the common formats, but we obviously can't be perfect for freeform text. The NumFocus projects have largely aligned, but there's also Google's format and ReST, which as been around forever, not to mention a few other entirely reasonable ways of documenting functions. We'd rather inject no information into analysis than incorrect information, so using doc strings here is potentially problematic.

So type hints are likely to show up; doc strings are a possibility.

[gnychis]

Agree with you @zooba ... with no standard on docstrings, I find type hinting to be far better off with PEP 484. Docstrings should at best just provide information about the function in the intellisense bubble.

[zooba]

@gnychis That last part is captured in #773 - definitely something we want to do.

[den-run-ai]

@zooba mypy seems to be working on support for docstring parsing for type annotations, so you could potentially get both done using their parser:

python/mypy#839

In general, why can't you use external parser for docstrings, like napoleon and sphinx?

[zooba]

@denfromufa That's a good idea. Currently we have a docstring parser for C extensions that I'd like to also use for Python files, but to be efficient it has to run as part of the completion DB refresh. This means that we can't really use it on open projects. But it could be designed to be swapped out for other parsers fairly easily.

I'm doing some experimental restructuring of our analysis engine (how we handle/track files, etc.) that I could work this into. It's my analysis-context branch if you want to follow along (note that it doesn't build yet, and there are basically just thousands of changes :) ).

[niijv]

Any update on the support for docstrings?

[EdStone]

Hi All:

What is the status of this please?

Thank you

[zooba]

We are still very keen to do this, but we don't have our roadmap planned out for the next release yet. Right now, still focused on getting our current release out.

[EdStone]

Hi zooba:

So ... still not clear.
Is it possible now to supply type hints (either through doc string or PEP484)?
If not, what is the best that we can actually do (in addition to using assert isinstance(a,int) )

Thank you,
E

[zooba]

@EdStone You can type them in, and you should see completions from the typing module, but they won't affect the completions we show. The assert isinstance pattern is still the only way to influence our analyser.

[EdStone]

Hi zooba:

Thank you for replying.

Unfortunately using asserts significantly clutters up code so its virtually useless.

Will you implement anything in the short-term that will allow me to provide type hints in separate files / skeletons (ex. in *.pyi files in some sort of parallel directory structure) in such a way that PTVS will pick up these skeletons and adjust autocomplete / intellisense accordingly.

In other words, replicate what PyCharm did with skeletons.

Thanks
E

[zooba]

@EdStone That will be part of our implementation for type hints, so it won't come any sooner than they do.

Currently you can write easy to analyse code in if False: blocks, but in the same file. We will ignore the condition and still read it, but it won't execute when you actually run your code. By easy to analyse I mean using obvious return types and calls:

if False:
    def my_func(a, b):
        return int()
    my_func(int(), int())

We'll pick these up and merge them with any other definitions, so it will work about the same as we expect type hints will.

The obvious downsides are all the things that type hints set out to fix, so I'm fully aware this is a workaround rather than a good solution. But it may help you get by while we work on it.

[EdStone]

Hi zooba:

Thank you for replying.

Yes, this approach is hacky so not sure if usable.

My main issues is that PTVS does not properly process external packages (ex. numpy, pandas, etc.), not my own code.

That's why I was asking about skeletons.
Do you think you can implement something similar to PyCharm's skeletons:
https://www.jetbrains.com/help/pycharm/2016.3/using-python-skeletons.html
This will definitely cause many people to convert to PTVS (PTVS is superb to all its competitors in all ways except intellisense which is probably most critical in Python).
Thanks,
E

[EdStone]

answer pls?

[gmonkman]

Keen to see this as well. Love using VS for my python as an old time supporter of MS dev products (since VB5). Don't want to have to jump ship to pycharm or similiar.