[CH1. Content Transfer-Encoding] No end-of-stream while collecting chunks

[vitorroriz]

On chapter 1 we are asked to implement content and transfer encoding. I'm handling gzip for content-encoding and chunked for transfer-encoding. My implementation seems to work fine for the pages I've tested against.

If transfer-encoding: chunked is present in the response headers I keep collecting the chunks to assemble the response body as a byte array. Given that response = s.makefile("rb", newline="\r\n"), being s our socket, the array composition is something like:

    def decodeTransfer(self, response, headers):
        assert headers["transfer-encoding"] == "chunked", "transfer-encoding: {}, not supported".format(headers["transfer-encoding"])

        data = bytearray() 
        while(True):
            line = response.readline()
            sz = int(line, 16) 
            if sz == 0:
                break
            chunk = response.read(sz)
            data += chunk
            crlf = response.readline()
        return data 

After that, if content-encoding: gzip is present in headers, I decode the array with gzip.decompress(). As I said, this works fine for the pages I've tested against. Except that, the page https://www.zggdwx.com/xiyou/1.html is used as example throughout chapter 2. For this page specifically, I keep getting EOFError: Compressed file ended before the end-of-stream marker was reached when trying to decompress the byte array with gzip lib.

From my understanding, that means that somehow I'm not getting or not adding the end-of-stream indication to the byte array. I'm trying to understand why.

[shuhei]

I couldn't find anything suspicious in your code. I also checked your repository and found that the error still happens even without the chunked encoding (tried with HTTP/1.0). I guess it's something to do with the website's gzip encoding and how it's decoded by gzip.decompress, but not sure...

[shuhei]

I don't know why, but zlib.decompressobj(16 + zlib.MAX_WBITS).decompress(data) seems to work while zlib.decompress(data, 16 + zlib.MAX_WBITS) doesn't 😅 . (Found at urllib3)

[vitorroriz]

Thank you for your help! I found an interesting thread and the reason that decompressobj works seems to be that it supports "partial" decompression, while zlib.decompress() will try to decompress the whole thing. It really seems that I'm not getting the whole stream, since even with decompressobj working, zlib.decompressobj has eof = False.

[shuhei]

Oh, I see! So the website's gzipped content misses some data at the end, probably gzip trailer.