StreamReader.iter_chunks() raises IndexError

[jlacoline]

Long story short

Any generator given by the iter_chunks() method on StreamReader will end raising an IndexError.
This comes from the readchunk() method which fails if the buffer is empty the end of file is reached.

Expected behaviour

stream.readchunk() should return b"" if the buffer is empty eof is reached
stream.iter_chunks() should return a generator that yields all the chunks and then raise StopAsyncIteration

Actual behaviour

stream.readchunk() raises IndexError if the buffer is empty eof is reached
stream.iter_chunks() returns a generator that yields all the chunks and then raises IndexError

Steps to reproduce

For stream.readchunk(): In the associated unit test in tests/test_streams.py, replace stream.read() with stream.readchunk()

    def test_readchunk(self):

        stream = self._make_one()

        def cb():
            stream.feed_data(b'chunk1')
            stream.feed_data(b'chunk2')
            stream.feed_eof()
        self.loop.call_soon(cb)

        data = self.loop.run_until_complete(stream.readchunk())
        self.assertEqual(b'chunk1', data)

        data = self.loop.run_until_complete(stream.readchunk())
        self.assertEqual(b'chunk2', data)

        data = self.loop.run_until_complete(stream.readchunk())
        self.assertEqual(b'', data)

Result:

――――――――――――――――――――――――――――――――――― TestStreamReader.test_readchunk ――――――――――――――――――――――――――――――――――――

self = <test_streams.TestStreamReader testMethod=test_readchunk>

    def test_readchunk(self):
    
        stream = self._make_one()
    
        def cb():
            stream.feed_data(b'chunk1')
            stream.feed_data(b'chunk2')
            stream.feed_eof()
        self.loop.call_soon(cb)
    
        data = self.loop.run_until_complete(stream.readchunk())
        self.assertEqual(b'chunk1', data)
    
        data = self.loop.run_until_complete(stream.readchunk())
        self.assertEqual(b'chunk2', data)
    
>       data = self.loop.run_until_complete(stream.readchunk())

tests/test_streams.py:565: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/loic/.pyenv/versions/3.5.3/lib/python3.5/asyncio/base_events.py:466: in run_until_complete
    return future.result()
/home/loic/.pyenv/versions/3.5.3/lib/python3.5/asyncio/futures.py:293: in result
    raise self._exception
/home/loic/.pyenv/versions/3.5.3/lib/python3.5/asyncio/tasks.py:239: in _step
    result = coro.send(None)
aiohttp/streams.py:329: in readchunk
    return self._read_nowait_chunk(-1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <StreamReader eof>, n = -1

    def _read_nowait_chunk(self, n):
>       first_buffer = self._buffer[0]
E       IndexError: deque index out of range

aiohttp/streams.py:363: IndexError

Your environment

• aiohttp master branch (identified on 2.2.3)
• Python 3.5.3
• Debian 8.8

[fafhrd91]

Fixed by pr

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.