Support for chunk_size

[tomchristie]

Closes #394

• Add iter_raw(chunk_size: int=None)
• Add iter_bytes(chunk_size: int=None)
• Add iter_text(chunk_size: int=None)

First bit of work towards chunk_size on the response.iter_[raw|bytes|text] methods,
using a decoder class in line with the other decoding.

Nice thing about this is that it's really easy to unit test, eg...

def test_byte_chunker():
    decoder = ByteChunker()
    assert decoder.decode(b"1234567") == [b"1234567"]
    assert decoder.decode(b"89") == [b"89"]
    assert decoder.flush() == []

    decoder = ByteChunker(chunk_size=3)
    assert decoder.decode(b"1234567") == [b"123", b"456"]
    assert decoder.decode(b"89") == [b"789"]
    assert decoder.flush() == []

    decoder = ByteChunker(chunk_size=3)
    assert decoder.decode(b"123456") == [b"123", b"456"]
    assert decoder.decode(b"789") == [b"789"]
    assert decoder.flush() == []

    decoder = ByteChunker(chunk_size=3)
    assert decoder.decode(b"123456") == [b"123", b"456"]
    assert decoder.decode(b"78") == []
    assert decoder.flush() == [b"78"]

[cdeler]

Are you sure about chunk_size with default None?

I do agree that it looks more suitable, but requests provides us with defaults chunk_size=1 or 512

[florimondmanca]

In this PR, when chunk_size=None we just return the input content unchanged, as one single big chunk.

Yes, this would deviate from what Requests seems to do, but:

• Setting a non-None default would break backward compatibility on our side.
• Defaulting to "transparently pass the chunk sent by the server" is probably the most reasonable approach anyway.

That said, we'd need to add this deviation from Requests to the compatibility guide. 👍

[florimondmanca]

Superb!

There may be room for factoring in some of the introduced logic within the _decoders.py module (bunch of repetition there), but happy to treat it as a follow-up.

Edit: so as per #1277 (comment), should we add a small note in our Requests compatibility guide on the default chunking behavior?

[florimondmanca]

In this PR, when chunk_size=None we just return the input content unchanged, as one single big chunk.

Yes, this would deviate from what Requests seems to do, but:

• Setting a non-None default would break backward compatibility on our side.
• Defaulting to "transparently pass the chunk sent by the server" is probably the most reasonable approach anyway.

That said, we'd need to add this deviation from Requests to the compatibility guide. 👍

[simonw]

I really like this - it's a great answer to my question in #1392 about safely retrieving a truncated version of a resource when dealing with a URL provided by an untrusted user, as an extra protection against denial-of-service on top of the HTTPX timeouts.

[florimondmanca]

And if I'm correct, this would close #394?

[piersoh]

Just clarify when it says "chunk_size=None we just return the input content unchanged, as one single big chunk" - presumably that means it returns each HTTP chunk (which may be big or small) as opposed to the whole response? How does this work for HTTP/2?

[tomchristie]

@piersoh Correct yes. For HTTP/2 that'll mean chunk sizes that correspond to the data frames on the wire.