-
-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow setting chunk_size
for Response.iter_bytes()
etc...
#394
Comments
I seem to remeber this playing into some primitives around streaming bytes vs text that we never ended up digging into? |
A good first pass onto this would be to change the decoder interface slightly, so that instead of eg. yeilding a byte chunk, they yield a list of byte chunks. On the first refactoring pass, we don't need to actually change the internal implmentation much - the decoders can just always yield a list with a single item. We'd then be able to add a |
Updated the issue title to reflect the current |
How could I help with this issue? |
Hi @b0g3r! I think this is still something we’d like to have, and given discussions in python-gitlab/python-gitlab#1036 it seems like some folks would like to see it too. :) Ways to move forward would be:
|
Do I understand correctly that we will need to forward chunk_size here? httpx/httpx/_dispatch/urllib3.py Lines 112 to 115 in a82adcc
|
@b0g3r Careful that we're in a sort of transition state w.r.t. urllib3 usage due to #804 (we'll soon use our own sync implementation, though keeping urllib3 as an option). Due to this I wouldn't advise relying on any existing urllib3 functionality — also because we'd want to provide chunk sizing on the async layer too, and it'd be odd to have a different implementation on both sides. I think we want to look at controlling the chunk size directly from |
@b0g3r So, as with comment #394 (comment) - the right place to start with this would be a pull request to https://github.com/encode/httpx/blob/master/httpx/_decoders.py that changes the interface of the decoders, so that they return a list of bytes rather than bytes. (And correspondingly, changing the places where the response calls the decoder such as Line 915 in a82adcc
I'd start with that as a foundational pull request, which will then make the remaining work much easier. (Adding chunk sizes to the decoder interface, and through to the response methods.) |
(a)iter_raw(self, chunk_size=1)
for part in self._raw_stream:
yield part let's use bytestring as buffer buffer = b""
for part in self._raw_stream:
buffer += part
while len(buffer) >= chunk_size:
yield buffer[:chunk_size]
buffer = buffer[chunk_size:]
if buffer:
yield buffer (a)iter_bytes(self, chunk_size=ITER_CHUNK_SIZE)
(a)iter_text(self, chunk_size=ITER_CHUNK_SIZE)
(a)iter_line(self, chunk_size=ITER_CHUNK_SIZE)
|
@tomchristie As I see |
Would be good to have |
Response.iter_bytes()
etc...
Response.iter_bytes()
etc...chunk_size
for Response.iter_bytes()
etc...
…es in whatever size the chunks are received (encode#394)
…es in whatever size the chunks are received (encode#394)
…es in whatever size the chunks are received (encode#394)
…es in whatever size the chunks are received (encode#394)
Requests allowed setting chunk_size within
.iter_content()
which is currently not an option for our alternatives.stream()
and.stream_text()
.For
.stream_text()
we should go the extra step and fix the issue that users sometimes run into when using this feature and use chunk-size for measuring the decoded text, not the raw bytes.The text was updated successfully, but these errors were encountered: