Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: maximum recursion depth exceeded when jsonpickle's version is too low #147

Closed
krisb78 opened this issue Mar 9, 2019 · 9 comments
Closed

RuntimeError: maximum recursion depth exceeded when jsonpickle's version is too low #147

krisb78 opened this issue Mar 9, 2019 · 9 comments
Assignees
@chanchiem
Labels
bug

Comments

@krisb78
Copy link

krisb78 commented Mar 9, 2019

Hello,

I want to give x-ray a go on django 1.11, but on this line:

aws-xray-sdk-python/aws_xray_sdk/core/models/entity.py

Line 254 in 707358c

return jsonpickle.encode(self, unpicklable=False)

I'm getting:

got an exception during serialization
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/models/entity.py", line 254, in serialize
    return jsonpickle.encode(self, unpicklable=False)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/__init__.py", line 313, in encode
    return json.encode(j.flatten(value))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 111, in flatten
    return self._pop([self.flatten(v) for v in obj])
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
   ...
  ...
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 111, in flatten
    return self._pop([self.flatten(v) for v in obj])
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 111, in flatten
    return self._pop([self.flatten(v) for v in obj])
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/pickler.py", line 227, in _flatten_key_value_pair
    if not util.is_picklable(k, v):
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/util.py", line 212, in is_picklable
    return not is_function(value)
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/util.py", line 176, in is_function
    if not is_object(obj):
  File "/usr/local/lib/python2.7/site-packages/jsonpickle/util.py", line 54, in is_object
    return (isinstance(obj, object) and
RuntimeError: maximum recursion depth exceeded in __instancecheck__

I'm assuming this is something I can fix in my code, but am hoping for a hint as to what it might be...
@chanchiem
Copy link
Contributor

Hey, thank you very much for reporting the issue. I have a few questions I'd like to ask you so that we can get to the root of this issue. Is this reproducible? If so, are you able to get a minimal block of code that can reproduce this? It would help us in troubleshooting it.

Also, what's your configuration like? Are you using anything other than the default configuration? (such as a custom context).

Is this being run on Lambda or on a local machine?

Thanks!

@krisb78
Copy link
Author

krisb78 commented Mar 12, 2019

Hi,

I tried to add it to existing project, so it's a bit hard for me to establish what the minimal block of code to reproduce this should be...

I really just modified my settings.py as follows.

  1. Add
'aws_xray_sdk.ext.django',

to INSTALLED_APPS

  1. Add
'aws_xray_sdk.ext.django.middleware.XRayMiddleware',

to MIDDLEWARE

  1. Add
XRAY_RECORDER = {
     'AWS_XRAY_DAEMON_ADDRESS': 'xray-daemon:2000',
     'AUTO_INSTRUMENT': True,  # If turned on built-in database queries and template rendering will be recorded as subsegments
     'AWS_XRAY_CONTEXT_MISSING': 'RUNTIME_ERROR',
     'PLUGINS': (),
	     'SAMPLING': True,
     'SAMPLING_RULES': None,
     'AWS_XRAY_TRACING_NAME': 'icourt', # the segment name for segments generated from incoming requests
     'DYNAMIC_NAMING': None, # defines a pattern that host names should match
     'STREAMING_THRESHOLD': None, # defines when a segment starts to stream out its children subsegments
 }

I should probably have mentioned that I'm using aws-xray-sdk==2.4.2

I'm test-running thins in docker on a cloud9 instance.

@krisb78
Copy link
Author

krisb78 commented Mar 12, 2019

Hello,

I did some more debugging on this and noticed that if I change the line of code that throws the exception to:

jsonpickle.encode(self, unpicklable=True)

the exception goes away and I get json output.

I then experimented a bit with the max_depth parameter to this, and noticed that the issue seem to be some kind of recursion in subsegments.

I'm a bit out of my element here, but perhaps this is related to the fact that I'm using multithreading (futures)?

Hope it helps.

@chanchiem
Copy link
Contributor

Hmmm as long as your threads are self contained and begin and end their own unit of work, it shouldn't cause any issues. What do your threads actually do? Is there a particular endpoint that causes this? It makes me very curious what's causing this to happen.

Is the segment being modified in any way? Usually this type of condition can happen if the entity contains an invalid parameter or it was some how corrupted. The fact that changing unpicklable=False to True may suggest that there's something that's corrupting it. The keyword is set to false to prevent extra information to be added to the serialized data.

@krisb78
Copy link
Author

krisb78 commented Mar 12, 2019 •
edited
Loading

The threads are used to fetch multiple pieces of data in parallel from a number of databases.
A ThreadPoolExecutor (https://pythonhosted.org/futures/) is initalised at startup, and then on some occasions multiple requests are made to in parallel to multiple dbs using futures.

I don't see any particular endpoint causing it.

I only got as far as modifying the settings file and making a test request to /, I don't do any custom instrumentation.

It's not intermittent and happens multiple times while a single request is handled.

@chanchiem
Copy link
Contributor

When you set your unpicklable to True, you mention that it seems to work. Is it possible for you to send over the raw trace information so that I can help diagnose this.

Since AUTO_INSTRUMENT is enabled in your Django app, does this issue still happen when it's disabled?

Just an extra precaution: Make sure you have the latest version of JsonPickle installed.

@krisb78
Copy link
Author

krisb78 commented Mar 13, 2019

Looks like upgrading jsonpickle did it! I was on 0.4, upgraded to 1.1 and the recursion error seems to have gone away.

I'll do some more testing probably in the evening, but it's looking good.

You might want to update the dependencies in setup.py.

Thanks for your quick responses on this @chanchiem , much appreciated!

@chanchiem
Copy link
Contributor

chanchiem commented Mar 13, 2019 •
edited
Loading

Good to know we were able to help you solve your problem :). I will add this as a task to do:

  • Attempt to reproduce this using JsonPickle version 0.4, and then slowly increment version number to see when this issue is resolved.
  • After finding the version, setup the version requirement for JsonPickle in the setup.py file.

@chanchiem chanchiem changed the title RuntimeError: maximum recursion depth exceeded in __instancecheck__ RuntimeError: maximum recursion depth exceeded when jsonpickle's version is too low Mar 13, 2019
@chanchiem chanchiem self-assigned this Mar 13, 2019
@chanchiem chanchiem added the bug label Mar 13, 2019
@krisb78 krisb78 mentioned this issue Mar 13, 2019
Closed
@srprash
Copy link
Contributor

srprash commented Dec 10, 2020

Since the jsonpickle dependency is unbounded it fetches v1.4.2. This should resolve the issue. Thanks!

@srprash srprash closed this as completed Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chanchiem chanchiem

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@krisb78 @chanchiem @srprash