asyncio.create_task() documentation should mention user needs to keep reference to the task

[vincentbernat]

BPO	44665
Nosy	@cmeyer, @nanjekyejoannah, @vincentbernat, @alexhartl
PRs	bpo-44665: asyncio.create_task() documentation should mention user needs to keep reference to the task #29163

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-18.06:01:26.261>
labels = ['3.11', 'type-feature', '3.9', '3.10', 'docs']
title = 'asyncio.create_task() documentation should mention user needs to keep reference to the task'
updated_at = <Date 2022-01-06.08:53:46.602>
user = 'https://github.com/vincentbernat'

bugs.python.org fields:

activity = <Date 2022-01-06.08:53:46.602>
actor = 'alexhartl'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2021-07-18.06:01:26.261>
creator = 'bernat'
dependencies = []
files = []
hgrepos = []
issue_num = 44665
keywords = ['patch']
message_count = 5.0
messages = ['397741', '404776', '404782', '404827', '409821']
nosy_count = 5.0
nosy_names = ['docs@python', 'cmeyer', 'nanjekyejoannah', 'bernat', 'alexhartl']
pr_nums = ['29163']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44665'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vincentbernat]

asyncio will only keep weak references to alive tasks (in _all_tasks). If a user does not keep a reference to a task and the task is not currently executing or sleeping, the user may get "Task was destroyed but it is pending!".

I would suggest adding the following paragraph to create_task() documentation:

Python only keeps weak references to the scheduled tasks. To avoid the task being destroyed by the garbage collector while still pending, a reference to it should be kept until the task is done.

And maybe an example in case a user wants something "fire and forget"?

running_tasks = set()
# [...]
task = asyncio.create_task(some_background_function())
running_tasks.add(task)
task.add_done_callback(lambda t: running_tasks.remove(t))

The same applies to ensure_future as it now uses create_task, so maybe a "See create_task()".

[nanjekyejoannah]

@bernat and ncoghlan, please see if the wording I have used in the linked PR helps to clarify this.

[cmeyer]

Is there a way to reproduce this issue? I run the following code in Python 3.9 and it works as expected (prints "xyz" twice).

import asyncio
import gc

async def xyz():
    print("xyz")

event_loop = asyncio.get_event_loop()

event_loop.create_task(xyz())

t = event_loop.create_task(xyz())
del t

gc.collect()

event_loop.stop()
event_loop.run_forever()

[vincentbernat]

Hummm, I have a hard time finding a short example when __del__ is not called until the task is finished. Sorry. I did run into this several time, for example here: ldo/dbussy#45 (and some internal projects as well). So, it happens from time to time, but it is hard to find a simple reproducer.

[alexhartl]

I just found this PR when a task of mine spontaneously crashed with a "Task was destroyed but it is pending" in the middle of program execution.

I think the warning should be added to loop.create_task(), too. Not sure if loop.call_later() and loop.call_at() are also affected?

I think it would be a good idea to add the fire-and-forget example that @bernat gave. At the moment, stackoverflow is full of suggestions to just use create_task() in this case, ignoring the return value. Actually, I think it is a true shortcoming that asyncio doesn't provide a fire-and forget functionality by itself.

[agrommek]

Hi everybody,

I noticed the warning about "you have to keep a reference to the task" at the python docs, but this triggered only more questions. Why should I need to keep a reference when the event loop also must have a reference to run the task? I even asked about it at Stack Overflow (see here) and only found this github issue later.

I agree with @alexhartl at #88831 (comment) that the workaround using add_done_callback() should be added to the docs. Moreover, I would suggest an addition to the warning, like so.

"Important Save a reference to the result of this function, to avoid a task disappearing mid execution. The event loop only keeps weak references to task objects."

Those two things together would have saved me some head-scratching.

Maybe for a later release of python (3.11?) an addition to the API for 'fire and forget tasks' would be nice, something like asyncio.create_task(coro, *, name=None, keep_strong_reference=False).

Thanks for your great work!

[nanjekyejoannah]

Thanks for clarifying, feel free to open a PR if you have time with the desired wording.

[agrommek]

I would like to contribute to the documentation of asyncio.create_task() and create a pull request. However, I have a (maybe stupid?) "never-have-contributed-to-cypthon-before" noob question: On which branch should I base the PR for the documentation on? Master branch (3.11 dev)? 3.7 where asyncio.create_task() was introduced? The development guide does not really say anything about this (or maybe I overlooked the information?).

Thanks!

[nanjekyejoannah]

@agrommek, Open on the main branch, the change can be backported to older branches if required.

[orodbhen]

I would be great, I think, if the documentation offered more explanation for this behavior. As it is, it just raises more questions, and I turned up several search results of people trying to make sense of it. When you read, your first thought is, "That can't be right. I must be missing something."

I agree with what's been said by others. This behavior is very non-intuitive, and adds to the difficulty of wrapping one's head around an already challenging paradigm. I do appreciate all the work that's being done to provide these tools, though. I expect these things will all get ironed out with time. Thanks to all.

[Pirulax]

The reason was mentioned above.

[dmoklaf]

I have just discovered this, after days hunting a bug where my tasks where disappearing. What is unclear to me now is:

1. Has all the asyncio library been checked for this issue while producing the documentation patch above? Or only the places identified in the issue's first post have been documented?

2. I am thinking in particular about the callback methods that return a Handle: call_soon, call_soon_threadsafe, call_later, call_at. If we don't keep the Handle reference in our program, can these callbacks be silently deleted (and thus, unscheduled) by the GC?

[SomberNight]

What about asyncio.run_coroutine_threadsafe? Does calling code need to keep a strong reference to the return value to avoid GC?

[amirsoroush]

Is there a way to reproduce this issue? I run the following code in Python 3.9 and it works as expected (prints "xyz" twice).

import asyncio
import gc

async def xyz():
    print("xyz")

event_loop = asyncio.get_event_loop()

event_loop.create_task(xyz())

t = event_loop.create_task(xyz())
del t

gc.collect()

event_loop.stop()
event_loop.run_forever()

Here is my answer for anyone wondering how to reproduce this issue (as their code works just fine without keeping a reference to the tasks). I tried to explain what I understood:

https://stackoverflow.com/a/76823668/13944524

[ggregoire]

@vincentbernat just to clarify, since I'm not that familiar with the intricacies of Python's garbage collector.

Is this enough (in the context of this issue) to keep a reference of the task like this?

async def coro():
    task1 = asyncio.create_task(...)
    task2 = asyncio.create_task(...)
    task3 = asyncio.create_task(...)
    task4 = asyncio.create_task(...)

Or is it really needed to keep them in a collection?

If so, can the collection be declared inside the coroutine? Or does it need to be outside to be a "strong" reference?

async def coro():
    running_tasks = set()
    task1 = asyncio.create_task(...)
    running_tasks.add(task1)
    task2 = asyncio.create_task(...)
    running_tasks.add(task2)
    task3 = asyncio.create_task(...)
    running_tasks.add(task3)
    task4 = asyncio.create_task(...)
    running_tasks.add(task4)

vs.

running_tasks = set()

async def coro():
    task1 = asyncio.create_task(...)
    running_tasks.add(task1)
    task2 = asyncio.create_task(...)
    running_tasks.add(task2)
    task3 = asyncio.create_task(...)
    running_tasks.add(task3)
    task4 = asyncio.create_task(...)
    running_tasks.add(task4)

Just trying to keep the code as simple as possible.

[vincentbernat]

Only the last one will work. The two other ones don't keep a reference on the tasks after the coro() function ends.

[ggregoire]

Only the last one will work. The two other ones don't keep a reference on the tasks after the coro() function ends.

But when the coro() function ends, all the tasks created inside coro() are done, so I don't need to keep the references anymore, right?

Maybe I should have clarified that I call the await inside the same coro() function.

So like:

async def coro():
    task1 = asyncio.create_task(...)
    task2 = asyncio.create_task(...)
    task3 = asyncio.create_task(...)
    task4 = asyncio.create_task(...)
    await task1
    await task2
    await task3
    await task4
    # all the tasks are done and coro() ends

In this case, would it work? Do I even need to keep the references task1, task2, etc or can I do directly away asyncio.create_task(...) without risking my tasks to be garbage collected before they are done?

async def coro():
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    await asyncio.create_task(...)
    # all the tasks are done and coro() ends

Thanks for the help.

---

Editing this post to avoid creating more noise:

I don't think this issue should be a support forum.

Agree but the documentation was updated following this issue and the wording is confusing. As you can see in the previous comments and in mine, it raises a lot of questions. The doc suggests to keep references of the tasks in any case, while apparently is not always necessary (like in my example).

I appreciate the help anyway. Thanks.

[vincentbernat]

I don't think this issue should be a support forum. But yes, both ways are fine (but they don't do the same thing).