Skip to content

Commit

Permalink
util.CallbackManager: keep strong references for running futures
Browse files Browse the repository at this point in the history
This clears up log spam for regtest tests.

related:
- https://bugs.python.org/issue44665
- python/cpython#88831
- https://textual.textualize.io/blog/2023/02/11/the-heisenbug-lurking-in-your-async-code/
- python/cpython#91887 (comment)
- "Task was destroyed but it is pending!"

Perhaps we should inspect all our usages of
- asyncio.create_task
- loop.create_task
- asyncio.ensure_future
- asyncio.run_coroutine_threadsafe
?

Example log for running a regtest test:
```
$ python3 -m unittest electrum.tests.regtest.TestLightningAB.test_collaborative_close
***** test_collaborative_close ******
initializing alice
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/home/user/wspace/electrum/electrum/logging.py", line 44, in format
    record = copy.copy(record)  # avoid mutating arg
  File "/usr/lib/python3.10/copy.py", line 92, in copy
    rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1781, in call_exception_handler
    self._exception_handler(self, context)
  File "/home/user/wspace/electrum/electrum/util.py", line 1535, in on_exception
    loop.default_exception_handler(context)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1744, in default_exception_handler
    logger.error('\n'.join(log_lines), exc_info=exc_info)
Message: "Task was destroyed but it is pending!\ntask: <Task pending name='Task-2' coro=<Abstract_Wallet.on_event_adb_set_up_to_date() running at /home/user/wspace/electrum/electrum/wallet.py:485> wait_for=<Future finished result=0> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.10/asyncio/futures.py:392]>"
Arguments: ()

[--- SNIP --- more of the same --- SNIP ---]

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/home/user/wspace/electrum/electrum/logging.py", line 44, in format
    record = copy.copy(record)  # avoid mutating arg
  File "/usr/lib/python3.10/copy.py", line 92, in copy
    rv = reductor(4)
ImportError: sys.meta_path is None, Python is likely shutting down
Call stack:
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1781, in call_exception_handler
    self._exception_handler(self, context)
  File "/home/user/wspace/electrum/electrum/util.py", line 1535, in on_exception
    loop.default_exception_handler(context)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1744, in default_exception_handler
    logger.error('\n'.join(log_lines), exc_info=exc_info)
Message: "Task was destroyed but it is pending!\ntask: <Task pending name='Task-31' coro=<Abstract_Wallet.on_event_adb_set_up_to_date() running at /home/user/wspace/electrum/electrum/wallet.py:485> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.10/asyncio/futures.py:385, Task.task_wakeup()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.10/asyncio/futures.py:392]>"
Arguments: ()
true
true
true
true
funding alice
```
  • Loading branch information
SomberNight committed Jul 17, 2023
1 parent 1767d26 commit ab05374
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion electrum/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -1706,6 +1706,7 @@ class CallbackManager:
def __init__(self):
self.callback_lock = threading.Lock()
self.callbacks = defaultdict(list) # note: needs self.callback_lock
self._running_cb_futs = set()

def register_callback(self, func, events):
with self.callback_lock:
Expand All @@ -1730,7 +1731,10 @@ def trigger_callback(self, event, *args):
for callback in callbacks:
# FIXME: if callback throws, we will lose the traceback
if asyncio.iscoroutinefunction(callback):
asyncio.run_coroutine_threadsafe(callback(*args), loop)
fut = asyncio.run_coroutine_threadsafe(callback(*args), loop)
# keep strong references around to avoid GC issues:
self._running_cb_futs.add(fut)
fut.add_done_callback(lambda fut_: self._running_cb_futs.remove(fut_))
elif get_running_loop() == loop:
# run callback immediately, so that it is guaranteed
# to have been executed when this method returns
Expand Down

0 comments on commit ab05374

Please sign in to comment.