Using Custom JSON Encoders and Decoders?

[erewok]

Hello,

I have recently discovered Starlette and am about to deploy a couple of small services using it and I have to say that I am pretty excited about this framework. I've been looking for the simple functionality provided by BackgroundTask for awhile now. Thanks so much for all the work that has gone into this project!

I am writing this issue because one of the things I typically do with HTTP APIs returning JSON is swap out the JSON libraries used for my own selection. I don't typically do this for performance (although it's a nice side benefit) so much as to control the ways in which JSON encoding and decoding happens. For example, I find that I'm dealing with UUIDs and datetimes a lot and it's nice to have them automatically encoded and decoded. I also tend to use rapidjson because it's quick and configurable (and it seems to see more development activity than ujson, which hasn't been updated in 3 years...)

Maybe an example would be helpful. In Starlette, I can construct a Response class like this:

import json
import typing

import rapidjson
from starlette.responses import JSONResponse


class CustomJSONEncoder(json.JSONEncoder):
    encode = rapidjson.Encoder(
        skip_invalid_keys=False,
        ensure_ascii=False,
        indent=None,
        sort_keys=False,
        number_mode=rapidjson.NM_NATIVE,
        datetime_mode=rapidjson.DM_ISO8601,
        uuid_mode=rapidjson.UM_CANONICAL)


class RapidJSONResponse(JSONResponse):
    media_type = "application/json"

    def render(self, content: typing.Any) -> bytes:
        return CustomJSONEncoder().encode(content).encode("utf-8")

And then, in a handler, I can use it like this:

async def index(request):
    conf = request.app.settings
    return RapidJSONResponse(
        {
            "status": http_helpers.Success.status,
            "version": conf.version,
            "test_uuid": uuid.uuid4(),
            "test_datetime": datetime.datetime.utcnow(),
            "message": f"Welcome to {conf.name}",
        }
    )

And everything just works:

In [1]: import requests

In [2]: requests.get("http://localhost:8000/").json()
Out[2]:
{'status': 'success',
 'version': 'local-dev',
 'test_uuid': 'c774a5d5-26c8-4292-9d42-a4e77dcd8a50',
 'test_datetime': '2019-11-13T05:10:21.610705',
 'message': 'Welcome to Starlette!'}

However, it's not easy to see how to use a custom decoder in Starlette. I am thinking that I could make an configurable parameter in the Starlette application object and pull it out in the Request object, but I'm not sure what you would say to that.

Q: Do you have suggestions for patching in a custom JSON decoder for JSON parsing in requests?

Q: Are you interested in a patch that allows users to patch in their own customer JSON decoder for requests?

[tomchristie]

At the moment you'll just need to parse request.body however you want.
In the future I think we'll have a generally configurable request parser, which'd let you configure not just JSON parsing, but any handle any content types, or override the multipart parser to handle upload files differently. (See #388)

[erewok]

Thanks for your reply. Your suggestion is a fine approach to my problem. As such, I'd be happy to close this ticket in favor of the other that you referenced. Alternately, we can it leave it open as another motivating use case.

Thanks again.

[aviramha]

At the moment you'll just need to parse request.body however you want.
In the future I think we'll have a generally configurable request parser, which'd let you configure not just JSON parsing, but any handle any content types, or override the multipart parser to handle upload files differently. (See #388)

How about adding a parameter to request_response with optional request class to use?
I guess there are more places to add it to support it, but generally speaking, is it a plausible feature?
(I'll be happy to send a PR)

[erewok]

I have been thinking about how to offer custom Request classes in Starlette, which would allow users to extend or modify all kinds of different behavior and solve this problem and others well. I had a couple of ideas for this, the first was using a type annotation to instruct Starlette to instantiate a custom request class instead of the standard. There are some things I like about this and some doubts I have. I want to get some feedback before pursuing.

[aviramha]

Typing annotation sounds good, but I'm not fully able to imagine how it would look like.

[erewok]

@aviramha @burritoIand I made a PR: #930

Feel free to comment on it.

[tomchristie]

How about adding a parameter to request_response with optional request class to use?

Perhaps. If we did do something like this then I'd expect it to look like...

• Add request_class = ... to the Starlette class API.
• Add request_class = ... to the Router class API. (Passed through from the app instance)

We'd also need to think carefully about what our expectations are wrt. middleware.

Personally I'd probably prefer to avoid this for the moment, and instead look into adding proper generic parsing support on the built-in Request instance.

[aviramha]

How about adding a parameter to request_response with optional request class to use?

Perhaps. If we did do something like this then I'd expect it to look like...

• Add request_class = ... to the Starlette class API.
• Add request_class = ... to the Router class API. (Passed through from the app instance)

We'd also need to think carefully about what our expectations are wrt. middleware.

Personally I'd probably prefer to avoid this for the moment, and instead look into adding proper generic parsing support on the built-in Request instance.

The middleware receives a subclass of Request, and any inheriting class shouldn't break LSP, so it shouldn't be a problem, no?

How do you imagine the generic parsing support on built-in Request instance?

[claydugo]

Is there a chance of revisiting this? I am interested in being able to use json_tricks for numpy array serialization.

Currently I am parsing the request.body as suggested above but it just adds extra steps to the process that do not feel necessary.

Looking here it seems like this step could be skipped altogether if you could specify a custom decoder.

This seems to be where all the relevant issues and PRs point to for discussion but I am sorry if that is not the case.

Thanks

[openSourcerer9000]

This is the source of the bug according to my stacktrace. I think it literally just needs to allow the user to pass cls down to this json.dumps.

...\lib\site-packages\starlette\responses.py", line 199, in render
    return json.dumps(
...

What is that, one line of code? Ya'll built this amazing server but can't finish a line of code in 4 years?

[bfontaine]

Instead of using a angry tone, make a PR with your solution.
(tip: it’s probably not one line of code)