Inspect getsource does not return the full source code for some decorated functions

[CodingYuno]

Bug report

The inspect getsource method fails to return the full source code of a target function if it has been decorated with a decorator that has been passed an argument with its own argument of a lambda function. This only happens if the decorator argument with the lambda is not the first argument of the decorator and there is an argument before it with a bracket e.g. a class or tuple.

from inspect import getsource


def decor(*args):
    def decorator(f):
        print(getsource(f))
    return decorator


@decor(dict(fun=lambda x: x+1))  # Works
def foo1():
    pass


@decor(dict(fun=lambda x: x+1), list())  # Works
def foo2():
    pass


@decor("1", 1, 0.1, [], {}, dict(fun=lambda x: x+1), list())  # Works
def foo3():
    pass


@decor(list(), dict(fun=lambda x: x+1))  # Fails
def foo4():
    pass


@decor((1, 2), dict(fun=lambda x: x+1))  # Fails
def foo5():
    pass


@decor((), (lambda x: x+1))  # Fails
def foo6():
    pass

The first three examples above will print the correct source code however the final three examples will print only:

@decor(list(), dict(fun=lambda x: x+1))


@decor((1, 2), dict(fun=lambda x: x+1)) 


@decor((), (lambda x: x+1))

The issue is caused by the inspect class BlockFinder wrongly setting self.indecorator to False when the token ")" is passed in the stream to tokeneater. This results in the lambda being able to incorrectly raise an EndOfBlock.

Potential Solution

By tracking if a "(" token has been ran whilst self.indecorator is True and preventing self.indecorator from going False until a ")" token is passed the issue is avoided.

The updated BlockFinder class which prevents this issue:

class BlockFinder:
    """Provide a tokeneater() method to detect the end of a code block."""
    def __init__(self):
        self.indent = 0
        self.islambda = False
        self.started = False
        self.passline = False
        self.indecorator = False
        self.decoratorhasargs = False
        self.last = 1
        self.body_col0 = None
        self.decorator_open_bracket = False
        self.decorator_args_open_bracket = 0

    def tokeneater(self, type, token, srowcol, erowcol, line):
        if not self.started and not self.indecorator:
            if token == "@":
                self.indecorator = True
            elif token in ("def", "class", "lambda"):
                if token == "lambda":
                    self.islambda = True
                self.started = True
            self.passline = True
        elif token == "(":
            if self.indecorator:
                self.decoratorhasargs = True
                if self.decorator_open_bracket:
                    self.decorator_args_open_bracket += 1
                else:
                    self.decorator_open_bracket = True
        elif token == ")":
            if self.indecorator and self.decorator_args_open_bracket:
                self.decorator_args_open_bracket -= 1
            elif self.indecorator:
                self.indecorator = False
                self.decorator_open_bracket = False
                self.decoratorhasargs = False
        elif type == tokenize.NEWLINE:
            self.passline = False
            self.last = srowcol[0]
            if self.islambda:
                raise EndOfBlock
            if self.indecorator and not self.decoratorhasargs:
                self.indecorator = False
        elif self.passline:
            pass
        elif type == tokenize.INDENT:
            if self.body_col0 is None and self.started:
                self.body_col0 = erowcol[1]
            self.indent = self.indent + 1
            self.passline = True
        elif type == tokenize.DEDENT:
            self.indent = self.indent - 1
            if self.indent <= 0:
                raise EndOfBlock
        elif type == tokenize.COMMENT:
            if self.body_col0 is not None and srowcol[1] >= self.body_col0:
                self.last = srowcol[0]
        elif self.indent == 0 and type not in (tokenize.COMMENT, tokenize.NL):
            raise EndOfBlock

Your environment

• CPython versions tested on: 3.10.8
• Operating system and architecture: Windows / Linux

[carljm]

Thanks for the careful debugging and the report. This was already fixed in #99654 and backported to 3.10 in fe7c309 -- if you upgrade to Python 3.10.10, it should contain the fix.