Content-Length header

[r7ar7a]

Long story short

I would need a simple way to remove the Content-Length header line from the multipart section of my POST request.

Expected behaviour

I'd expect this line to ensure that.
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/formdata.py#L133

Actual behaviour

This line puts the header back anyway.
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/multipart.py#L789

Steps to reproduce

import asyncio
import aiohttp
from aiohttp import ClientSession, ClientError
from io import BytesIO, StringIO

from aiohttp.multipart import MultipartWriter
from aiohttp.formdata import FormData 

loop = asyncio.get_event_loop()
session = ClientSession(loop=loop)

class W(object):
    def __init__(self, writer):
        self.writer = writer
        
    async def write(self, *args, **kwargs):
        self.writer.write(*args, *kwargs)
        
    async def getvalue(self):
        return self.writer.getvalue()
    
fd = FormData(charset='utf-8')
fd.add_field('asd123', StringIO('asd111'), content_transfer_encoding='binary')
writer = fd()
async def x():
        y = W(BytesIO())
        await writer.write(y)
        print('=============')
        print((await y.getvalue()).decode('utf8'))
        print('=============')
a = loop.run_until_complete(x())

Your environment

aiohttp==2.2.3

[hellysmile]

#379

[r7ar7a]

Thanks for the quick reply, but that doesn't help me this case, because the CONTENT_LENGTH header is explicitly added when calling aiohttp.multipart.MultipartWriter.append_payload(self, payload) and right after the header string is generated here:
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/multipart.py#L793

Am I missing something? Or did I misunderstood your answer? I tried this:
session = ClientSession(skip_auto_headers=[CONTENT_LENGTH], loop=loop)
It didn't help.

[r7ar7a]

Be aware that I'm talking about the header of the multipart section:

POST /?a=2 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Host: localhost:2000
User-Agent: Python/3.6 aiohttp/2.2.3
Content-Length: 421
Content-Type: multipart/form-data; boundary="444afd8e74d3423c9aff8c525be99f87"

--444afd8e74d3423c9aff8c525be99f87
Content-Disposition: form-data; name="alma"; filename="alma"; filename*=utf-8''alma
Content-Type: text/plain; charset=utf-8
THIS ONE HERE
ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇ
Content-Length: 5
^^^^^^^^^^^^^^^^
korte
--444afd8e74d3423c9aff8c525be99f87

[asvetlov]

What is the point for header removing?

[r7ar7a]

Sorry, my bad.
I experienced that if I remove all Content-Length header, my db server accepts the query and parses it successfully. But the problem was this issue: #2066.

[r7ar7a]

... can be closed

[FichteFoll]

While this isn't a bug per se, I'd still be interested as to why a Content-Length header is added for each part since this isn't specified in any RFC¹. In fact, there was even call to remove bad examples in SIP-related RFCs, where this was used in examples².

¹ RFCs I checked: 1867, 2388, 7230, 7231, 7233

² https://www.ietf.org/mail-archive/web/sipcore/current/msg04739.html https://www.rfc-editor.org/errata_search.php?rfc=6086&eid=3167

[asvetlov]

@kxepal can you shed a light?

[kxepal]

Well, it's simple.

1. Because we can (: It's a good hint for your parser to know how much data to read before part is over. Semantic of Content-Length is pretty known and standard.

2. While none of RFC allows to do that, none one forbids to do that. Unexpectable headers should be ignored per RFC.

3. Originally, this behavior was borrowed from CouchDB which sends multipart body parts with attachments with Content-Length header set. From CouchDB side this is a way to preserve attachment meta information which includes blob`s number of bytes. So your client/server can do the same.

Probably, we can drop it and feel no sorry, but imagine the case you receive multipart payload with some text and images and you need to decide to do preview or not depending on their size. Much likely you wouldn't let do preview for 3+ MiB images, while small one you would like to. Content-Length header could help you to make a decision long before you start read body part data. You can even skip it with easy - you know how long you should leap to start read the next one.

So, it's very helpful hint for your clients and servers. You can ignore it if you like, but when you'll need it - it'll be there. I think 14+X{1-6} additional bytes per body part is not a big price for such opportunities.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new