Fix strong references to mutable objects in context.clone

[nelfin]

Steps

• For new features or bug fixes, add a ChangeLog entry describing what your PR does.
• Write a good description on what the PR does.

Description

Since InferenceContext.clone passes self.path when
constructing a new instance, these two objects will hold references to
the same underlying set object. This caused modifications to
context.path further down in the stack to be reflected in "earlier"
objects and this would incorrectly detect a loop if the same node was
used twice in the inference of a some node (instead of only if an
inference result was used in inferring itself).

Type of Changes

	Type
✓	🐛 Bug fix
	✨ New feature
	🔨 Refactoring
	📜 Docs

Related Issue

Closes #926

[nelfin]

This test has changed behaviour. I assume that it's okay for it to raise InferenceError instead of returning Uninferable as long as it still doesn't leak StopIteration

[hippo91]

It seems fair.

[hippo91]

@cdce8p what is your opinion?

[cdce8p]

I'm not too familiar with that part of the code base. If the pylint tests continue to pass, I would guess it's fine.

[Pierre-Sassoulas]

Potentially this could cause new crash if an inference error is raised in calling code ? I don't think pylint tests cover all possible edge case.

[nelfin]

Updated and rebased to appease code formatting overlords

[hippo91]

@nelfin it is a great improvment. In fact you succeeded where i failed with #884. Congrats!
I just left a bunch of comments.
I also tried the tests of pylint with your astroid branch and one test is failing. It deals with logging module and it is possible that the warnings that are now emitted are correct. Could you please have a look to it? @Pierre-Sassoulas can you also have a look? I saw you just reintroduced the concerned lines, so you will be much pertinent than me.

[hippo91]

It seems fair.

[nelfin]

I also tried the tests of pylint with your astroid branch and one test is failing. It deals with logging module and it is possible that the warnings that are now emitted are correct. Could you please have a look to it?

I think the new messages are okay? There wasn't any warning previously raised for the following:

class Logger(renamed_logging.Logger):
    pass

custom_logger = Logger('three')
custom_logger.info('testing {0}'.format('info'))  # here
custom_logger.info('testing %s' % 'info')   # and here

but those newly raised warning match the behaviour you would expect if Logger was being correctly inferred now. I'll update that test and submit a PR to pylint

[hippo91]

@nelfin i will wait for @Pierre-Sassoulas remarks concerning the pylint test before merging this. Thanks again.

[nelfin]

@nelfin i will wait for @Pierre-Sassoulas remarks concerning the pylint test before merging this.

No problem.

Thanks again.

Thank you!

[cdce8p]

@hippo91 I just looked at pylint-dev/pylint#4325 and also ran the other functional tests. The error with the logging tests is indeed a false-negative currently. Since all seems to work as expected then, I would say this is good to go.

[Pierre-Sassoulas]

👍

[cdce8p]

@Pierre-Sassoulas mentioned here that the change from Uninferable to StopIteration might have unintended consequences. After I saw it, I ran the changes against one of my repos and indeed saw multiple errors. Namely: not-callable, unsupported-membership-test, and invalid-sequence-index. I haven't investigated what code exactly is causing these, but it's safe to assume this shouldn't be merged just yet.

The Github Actions run: https://github.com/cdce8p/ha-core/runs/2308362444?check_suite_focus=true#step:7:58

[cdce8p]

I had some time to look at two of them and reduced it down to code snippets. Maybe that helps.

unsupported-membership-test

# pylint: disable=missing-docstring,too-few-public-methods,invalid-name
from enum import Enum

class MyEnum(Enum):
    CONST1 = "const"

def name_in_enum(name):
    # false-positive: unsupported-membership-test
    if name in MyEnum.__members__:
        return

not-callable

pip install aiohttp python-telnet-vlc

# pylint: disable=missing-docstring,unused-import,protected-access
from aiohttp import web
from python_telnet_vlc import VLCTelnet

def func():
    app = web.Application()
    app._router.freeze = lambda: None

class VlcDevice:
    def __init__(self):
        self._vlc = VLCTelnet("VLC-TELNET", "pwd", "0000")

    def media_previous_track(self):
        # False-positive: not-callable
        self._vlc.prev()

    def media_next_track_abc(self):
        # False-positive: not-callable
        self._vlc.next()

[nelfin]

I had some time to look at two of them and reduced it down to code snippets. Maybe that helps.

Thanks for that, it helps a lot.

I spent some time trying to replicate these messages from your bulid log and debugging the causes. I picked up these errors from the log in the link:

Error: homeassistant/components/vlc_telnet/media_player.py:289:8: E1102: self._vlc.prev is not callable (not-callable)
Error: homeassistant/components/vlc_telnet/media_player.py:293:8: E1102: self._vlc.next is not callable (not-callable)
Error: homeassistant/components/mpd/media_player.py:393:14: E1102: self._client.next is not callable (not-callable)
Error: homeassistant/components/ezviz/binary_sensor.py:26:23: E1135: Value 'BinarySensorType.__members__' doesn't support membership test (unsupported-membership-test)
Error: homeassistant/components/ezviz/sensor.py:26:23: E1135: Value 'SensorType.__members__' doesn't support membership test (unsupported-membership-test)
Error: homeassistant/components/xmpp/notify.py:193:20: E1126: Sequence index is not an int, slice, or instance with __index__ (invalid-sequence-index)

The not-callable messages stem from the fact that the previous behaviour was to infer [<Const.None>, <BoundMethod.add>] but after the application of this change it only infers [Const.None]. I haven't yet determined the cause and hopefully a smaller snippet will help.

The Enum.__members__ seems to be a false-positive. I'll see why, given that it should probably be inferred as [''] via brain_namedtuple_enum. I reduced the example to an almost identical snippet to yours.

The invalid-sequence-index from xmpp/notify.py:193 is interesting. It is newly inferred, the type of message was not inferred before. Inferring message gets us an instance of xmpp.Message, but inferring the Subscript gets us a list since bases.Instance.getitem returns the first inferred result from infer_call_result on __getitem__. In the general case, we should probably update infer_getitem to return all values so that pylint/safe_infer doesn't return (maybe extend _check_invalid_sequence_index to check that all inferred values are sequence types, like how no-member only emits if none of the inferred values have that attribute). In the specific case of xmpp.Message/ElementBase, there's potential for a brain to be added to return Uninferable because the behaviour of __getitem__ can change dynamically at runtime due to loaded plugins. Here is a minimal example that matches the behaviour that I consider to be a false-positive:

class DynamicGetitem:
    def __getitem__(self, key):
        if key == 'attributes':
            return []
        return {'world': 123}


ex = DynamicGetitem()
ex['hello']['world']
#  E1126: Sequence index is not an int, slice, or instance with __index__ (invalid-sequence-index)

[nelfin]

The Enum.__members__ seems to be a false-positive. I'll see why, given that it should probably be inferred as [''] via brain_namedtuple_enum. I reduced the example to an almost identical snippet to yours.

This wasn't quite correct. members = [''] is for enums built via Enum.__call__, e.g. Foo = Enum('Foo', 'bar baz').

So it looks like this has uncovered an existing bug: before the patch the value of SensorType.__members__ was Uninferable so the membership test check was skipped, after the patch __members__ is inferred to a Property and not transformed to the expected value of that property. Here's an example repro that fails with pylint 2.7.4 on current astroid master f2b197a:

# enum_repro.py

class BaseMeta(type):
    def __new__(metacls, cls, bases, classdict):
        classdict['__members__'] = metacls.__members__
        return super().__new__(metacls, cls, bases, classdict)

    @property
    def __members__(cls):
        return {'hello', 'world'}


class Parent(metaclass=BaseMeta):
    pass


class Derived(Parent):
    pass


assert 'hello' in Parent.__members__
assert 'world' in Derived.__members__

And example output from astroid:

# enum_repro.py

import astroid
mod = astroid.MANAGER.ast_from_file('enum_repro_mod.py', 'mod', source=True)
Parent, Derived = mod.body[1], mod.body[2]
print(list(Parent.igetattr('__members__')))
# [<Set.set l.10 at 0x...>]
print(list(Derived.igetattr('__members__')))
# [<Property.__members__ l.8 at 0x...>]

[nelfin]

@cdce8p: aiohttp and VLCTelnet turned out to be red herrings. This case fails on current stable versions:

class Example:
    def prev(self):
        pass
    def next(self):
        pass
    def other(self):
        pass


ex = Example()
ex.other()  # no warning
ex.prev()   # no warning
ex.next()   # no warning

import typing

ex.other()  # no warning
ex.prev()   # false-positive: not-callable
ex.next()   # false-positive: not-callable

[nelfin]

I've bisected this down to 78d5537. Pylint 2.3.1 passes this case with 20a7ae5 and fails with 78d5537

[cdce8p]

Ok, let's merge it