Streaming a database CSV export

[fxn]

I read in the docs you can pass a file-like object to HTTPX for doing multipart file uploads. I guess that is what I am looking for in my use case, but don't know quite how to connect the docs (new to Python too).

My data comes from a CSV export being consumed from MySQL:

with mysql.connector.connect(database=database) as connection:
    with connection.cursor(dictionary=dictionary) as cursor:
        cursor.execute('SELECT * FROM table')
        csv.writer(tmp).writerows(cursor)
            

Right now, I am storing the CSV export in an intermediate file, and then doing a file upload.

This works, but these exports can be in the several GBs and ideally I'd wrap that cursor in something that yields CSV lines or whatever, pass that object to HTTPX, and have the upload consume the result set by itself. Should be streamed as chunked encoding, since we do not know Content-Length beforehand.

Is there a way to accomplish this? If yes, which exact interface should such wrapper provide?

[Bluenix2]

It is mentioned very quickly in Sending Binary Request Data - QuickStart. You can give HTTPX a generator that yields bytes.

The next problem is how you actually structure this. Sadly, there's not much we can do in terms of reducing copying of memory because of how csv constructs new strings and doesn't allow us to re-use memory from what I know. I would use a utility-class like this:

from queue import Queue
from typing import Generator, Optional, Type
from types import TracebacType


class ReadWriteQueue:
    def __init__(self, buffer_size: int) -> None:
        self._queue = Queue(buffer_size)

    def __enter__(self) -> Self:
        return self

    def __exit__(
        self,
        exc_type: Optional[Type[BaseException]],
        exc_val: Optional[BaseException],
        traceback: Optional[TracebackType]
    ) -> None:
        self._queue.put('')

    def read(self) -> str:
        return self._queue.get()

    def write(self, item: str) -> None:
        self._queue.put(item)


def yield_rwqueue_bytes(queue: ReadWriteQueue) -> Generator[bytes, None, None]:
    chunk = queue.read()
    while chunk:
        yield chunk.encode('utf-8')
        chunk = queue.read()

To fit this into the code you posted, it would look like the following:

with mysql.connector.connect(database=database) as connection:
    with connection.cursor(dictionary=dictionary) as cursor:
        # I recommend a buffer of one, since you can write multiple
        # rows in one go anyways
        with ReadWriteQueue(1) as q:
            writer = cs.writer(q)
            # Start up a thread with the HTTPX request where
            # content=yield_rwqueue_bytes(q)
            ...

            cursor.execute('SELECT * FROM table')
            writer.writerows(cursor)

That said, the example you passed doesn't make much sense with the cursor. If the data is so big wouldn't you have multiple .writerows() calls? I assume this was a simplification.

If that helped, don't forget to mark this as answered.

[Bluenix2]

I see HTTPX tries to read the entire file to compute Content-Length here by calling read() once. So, we get a Content-Length, and only foo is sent.

The reason this happens is because your MinimalIO() class is incorrectly implemented. HTTPX assumes that MinimalIO() is a file opened with open() which would be read to the end upon calling read() (which means that you should consume the whole iterator: ''.join(self.iter)).

Do you know if multipart file upload with chunked encoding is possible?

Looking at the code, yes it appears that HTTPX insists on computing a Content-Length header unlike when you pass an iterable to content and Transfer-Encoding gets set to chunked. This may have been an oversight seeing as RFC 1867 doesn't explicitly say that Content-Length is required:

5.2 Deferred file transmission

In some situations, it might be advisable to have the server validate
various elements of the form data (user name, account, etc.) before
actually preparing to receive the data. However, after some
consideration, it seemed best to require that servers that wish to do
this should implement this as a series of forms, where some of the
data elements that were previously validated might be sent back to
the client as 'hidden' fields, or by arranging the form so that the
elements that need validation occur first. This puts the onus of
maintaining the state of a transaction only on those servers that
wish to build a complex application, while allowing those cases that
have simple input needs to be built simply.

The HTTP protocol may require a content-length for the overall
transmission. Even if it were not to do so, HTTP clients are
encouraged to supply content-length for overall file input so that a
busy server could detect if the proposed file data is too large to be
processed reasonably and just return an error code and close the
connection without waiting to process all of the incoming data. Some
current implementations of CGI require a content-length in all POST
transactions.

[...]

I think the next step might be opening an issue and/or PR, however I'd like to hear @tomchristie's opinion on this.

[fxn]

The reason this happens is because your MinimalIO() class is incorrectly implemented. HTTPX assumes that MinimalIO() is a file opened with open() which would be read to the end upon calling read() (which means that you should consume the whole iterator: ''.join(self.iter)).

Yes, yes, that code was a poor attempt at trying to show one attempt that didn't work, because it implements something that does not match the current API.

Chunked encoding has nothing to do with the Content-Type. It goes even at a lower level than compression. If your request is compressed, the compressed bytes are chunked. In the other end, you first process the chunks, and then the compression, and all that yields the real bytes to the high-level code that processes the request body.

[fxn]

So, if you folks confirm the answer to my original question is: "no, you cannot upload a file from a generator/iterator using multipart and chunked encoding today", we could close this, and then perhaps followup in an issue?

[fxn]

Let me show you curl(1) doing this:

% cat foo.txt
test
% curl -H "Transfer-Encoding: chunked" -F file=@foo.txt http://localhost:3000

The server gets

POST / HTTP/1.1
Host: localhost:3000
User-Agent: curl/7.79.1
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=------------------------541cf4e4e4891f84

be
--------------------------541cf4e4e4891f84
Content-Disposition: form-data; name="file"; filename="foo.txt"
Content-Type: text/plain

test

--------------------------541cf4e4e4891f84--

0

You see, this is a multipart/form-data body chunked, like you could chunk text/plain, Content-Type does not matter.

[Bluenix2]

So, if you folks confirm the answer to my original question is: "no, you cannot upload a file from a generator/iterator using multipart and chunked encoding today", we could close this, and then perhaps followup in an issue?

To be clear I am no representative for this project, but no I don't think this is possible.

[fxn]

This seems to be unsupported as of this writing. I have opened #2249 for it.

Thanks all!