Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RetryError cannot be serialized with pickle #182

Closed
mezgerj opened this issue Jul 18, 2019 · 2 comments · Fixed by #183
Closed

RetryError cannot be serialized with pickle #182

mezgerj opened this issue Jul 18, 2019 · 2 comments · Fixed by #183

Comments

@mezgerj
Copy link
Contributor

mezgerj commented Jul 18, 2019

Currently, RetryError is unpickleable. To ensure that exceptions raised from the tenacity library are pickle-able they should provide the original arguments it was instantiated with in its .args attribute.This affects the ability to use tenacity inside of a celery task for example. I am working on a PR for this issue.

@crdoconnor
Copy link

crdoconnor commented Sep 18, 2019

This is causing me headaches too. It makes tenacity not usable with celery.

@mergify mergify bot closed this as completed in #183 Oct 28, 2019
@Alexander3
Copy link

Alexander3 commented Oct 8, 2021

I still have this issue using tenacity==7.0.0 and celery==4.4.7
I tried updating, but I get the same error in tenacity==8.0.1

from celery import shared_task
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
)

@retry(
    retry=retry_if_exception_type(ValueError), stop=stop_after_attempt(1),
)
def foo():
    raise ValueError()


@shared_task()
def test_exp():
    foo()

Calling this task results in

[2021-10-08 19:17:51,368: ERROR/ForkPoolWorker-7] Task apps.main.tasks.test_exp[539feed0-0c87-4a61-ad2a-ba158e7854a6] raised unexpected: RetryError('<Future at 0x7fb0ac4797f0 state=finished raised ValueError>')
Traceback (most recent call last):
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 197, in _inner
    reraise(*exc_info)
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 54, in reraise
    raise value
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 192, in _inner
    return f(*args, **kwargs)
  File "/home/bond/work/current/nadii/NadiiUI/backend/apps/main/tasks.py", line 151, in test_exp
    foo()
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/tenacity/__init__.py", line 339, in wrapped_f
    return self(f, *args, **kw)
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/tenacity/__init__.py", line 430, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in iter
    six.raise_from(retry_exc, fut.exception())
  File "<string>", line 3, in raise_from
tenacity.RetryError: <Future at 0x7fb0ac4797f0 state=finished raised ValueError>
[2021-10-08 19:17:51,378: ERROR/MainProcess] Task handler raised error: <MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fb0ac4797f0 state=finished raised ValueError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.>
Traceback (most recent call last):
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/billiard/pool.py", line 366, in workloop
    put((READY, (job, i, result, inqW_fd)))
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/billiard/queues.py", line 366, in put
    self.send_payload(ForkingPickler.dumps(obj))
  File "/home/bond/.virtualenvs/backend-4I_U3jTL/lib/python3.8/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
billiard.pool.MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fb0ac4797f0 state=finished raised ValueError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.

sergei-maertens added a commit to open-formulieren/open-forms that referenced this issue Apr 8, 2024
The retry error from tenacity uses threading locks under the hood,
which cannot be pickled. Specifying reraise=True avoids this
particular codepath.

See upstream issues:

* jd/tenacity#429
* jd/tenacity#182
* jd/tenacity#147
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants