Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables #88342

Closed
alexdelorenzo mannequin opened this issue May 19, 2021 · 3 comments
Labels
stdlib Python modules in the Lib dir topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@alexdelorenzo
Copy link
Mannequin

alexdelorenzo mannequin commented May 19, 2021

BPO 44176
Nosy @asvetlov, @1st1, @alexdelorenzo
PRs
  • bpo-44176: Allow asyncio.as_completed()'s first parameter to be a generator yielding awaitables #26228
  • Files
  • as_completed_gen.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-05-19.05:04:33.179>
    labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library', 'expert-asyncio']
    title = 'asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables'
    updated_at = <Date 2021-05-19.05:53:09.349>
    user = 'https://github.com/alexdelorenzo'

    bugs.python.org fields:

    activity = <Date 2021-05-19.05:53:09.349>
    actor = 'alexdelorenzo'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)', 'asyncio']
    creation = <Date 2021-05-19.05:04:33.179>
    creator = 'alexdelorenzo'
    dependencies = []
    files = ['50052']
    hgrepos = []
    issue_num = 44176
    keywords = []
    message_count = 1.0
    messages = ['393923']
    nosy_count = 3.0
    nosy_names = ['asvetlov', 'yselivanov', 'alexdelorenzo']
    pr_nums = ['26228']
    priority = 'normal'
    resolution = None
    stage = None
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue44176'
    versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

    Linked PRs

    @alexdelorenzo
    Copy link
    Mannequin Author

    alexdelorenzo mannequin commented May 19, 2021

    According to the documentation, asyncio.as_completed() takes a positional argument, aws, as an iterable of awaitables[1]:

        asyncio.as_completed(aws, *, loop=None, timeout=None)
          Run awaitable objects in the aws iterable concurrently.

    As seen in the attached as_completed_gen.py file, built-in containers like lists, and iterators over them, are accepted by as_completed() as the first parameter without raising an error.

    However, as_completed() raises TypeError if it is called with an iterable of awaitables that is also a generator. There are examples of this behavior in as_completed_gen.py, but here is a short example using a generator expression in the main() coroutine function:

        from asyncio import run, as_completed
        
        async def example(): pass
        
        async def main():
          coros = (example() for _ in range(10))
        
          for coro in as_completed(coros):  # raises TypeError
            await coro
        
        run(main())

    Running that example will raise a TypeError with this message:

    TypeError: expect an iterable of futures, not generator
    

    If we look at the first line in the body of as_completed(), we can see why this error is thrown for generators that yield awaitables:

        def as_completed(fs, *, loop=None, timeout=None):
          if futures.isfuture(fs) or coroutines.iscoroutine(fs):
            raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
          ...

    Because generators are coroutines, and the first condition in as_completed() is True, and TypeError gets raised:

        from asyncio import coroutines
    # generators and generator expressions are coroutines
    assert coroutines.iscoroutine(example() for _ in range(10))
    

    Perhaps as_completed() can use inspect.isawaitable() instead, like so:

        from inspect import isawaitable
    
        def as_completed(fs, *, loop=None, timeout=None):
          if futures.isfuture(fs) or isawaitable(fs):
            ...

    I made a pull request with that change here[2].

    [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

    [2] #26228

    @alexdelorenzo alexdelorenzo mannequin added 3.7 (EOL) end of life 3.8 (EOL) end of life 3.9 only security fixes 3.10 only security fixes topic-asyncio type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir labels May 19, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @ezio-melotti ezio-melotti moved this to Todo in asyncio Jul 17, 2022
    @kumaraditya303 kumaraditya303 removed 3.10 only security fixes 3.9 only security fixes 3.8 (EOL) end of life 3.7 (EOL) end of life labels Oct 18, 2022
    @kumaraditya303
    Copy link
    Contributor

    3.12 allows generators so this is completed.

    @kumaraditya303
    Copy link
    Contributor

    #103626 clarifies this in doc.

    carljm added a commit to carljm/cpython that referenced this issue Apr 20, 2023
    * main: (24 commits)
      pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561)
      pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545)
      pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634)
      pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633)
      pythongh-102856: Initial implementation of PEP 701 (python#102855)
      pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589)
      pythongh-83004: Harden msvcrt further (python#103420)
      pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626)
      pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314)
      pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586)
      pythongh-103583: Always pass multibyte codec structs as const (python#103588)
      pythongh-103617: Fix compiler warning in _iomodule.c (python#103618)
      pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600)
      pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576)
      pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613)
      [Doc] Fix a typo in optparse.rst (python#103504)
      pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531)
      pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039)
      pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569)
      pythongh-67230: update whatsnew note for csv changes (python#103598)
      ...
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir topic-asyncio type-bug An unexpected behavior, bug, or error
    Projects
    Status: Done
    Development

    No branches or pull requests

    1 participant