deficiency in new webhook implementation

[ghost]

Issue type

[ ] Feature request
[X] Bug report
[ ] Documentation

Environment

• Python version: python 2.7.5
• NetBox version: develop-2.4

Description

Testing out the webhook implementation and discovered the following bug:
when a model contains a custom field of type date the worker is unable to serialize the data for transmission

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rq/worker.py", line 793, in perform_job
    rv = job.perform()
  File "/usr/lib/python2.7/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/usr/lib/python2.7/site-packages/rq/job.py", line 605, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/opt/netbox/netbox/extras/webhooks_worker.py", line 44, in process_webhook
    prepared_request = requests.Request(**params).prepare()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 259, in prepare
    hooks=self.hooks,
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 307, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 427, in prepare_body
    body = json_dumps(json)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 268, in default
    o.__class__.__name__)
TypeError: Object of type date is not JSON serializable

[lampwins]

@ridoline I am unable to reproduce this with a Date custom field on the Device model. Please provide the exact steps necessary to raise this issue.

[ghost]

@lawpwins, I suppose it might have been an issue with my data or has been resolved, as I recently refreshed my development box with production data and the issue is no longer manifesting itself.

[lampwins]

I played with this some last night and was able to raise the issue a handful of times, but I have yet to find a root cause or even a reliable way of triggering the issue. I will be digging into this more.

[lampwins]

I finally ran this down. There are two separate issues in play. First, date type custom field values are currently actual datetime objects and the default json serializer in the requests library doesn't handle this well. I am still a little unclear why DRF is okay with this, since we are using the same API serializers. Second, the current signaling system for Webhooks suffers from the same problems of related objects (custom fields and tags) not being associated at the time of the post_save signal, as the original Change Logging implementation. This was addresses in Change Logging in df1f339. I think it makes a lot of sense to wrap the Webhook signaling into the ChangeLoggingMiddleware since the mechanics of both features are very similar.

[lampwins]

Just to be clear, this was broken out from #2282 and thus this issue still needs to be addressed.

[lampwins]

I finally nailed the root cause down. DRF processes the JSON encoding after the serialization, so by manually using the serialisers we miss the DRF specific encoding which makes use of there own JSON encoder class which correctly handles datatime objects (and many others) https://github.com/encode/django-rest-framework/blob/7b1582e00e91001ec07cd394520ec5cdd2c2add1/rest_framework/utils/encoders.py#L34. Currently we are using the built-in JSON encoder of the requests library, so we simply need to specify an alternate encoder to fix this.