load big files in chunks

[mgckind]

No description provided.

[kadrlica]

Would it make sense for this function to live in dtypes.py?

[mgckind]

Yes, although here is where those file_type are defined. Don't mind either way

[kadrlica]

Is there a downside to loading by chunks? Should we always be doing it? What about a default chunksize set in the ea_config.py (maybe set to a fairly large value)?

[mgckind]

I didn't see any. if not chunksize is defined the default works as fast as the previous version. Using a default chunksize might work as well but its a tricky number. Today I uploaded 25M 4-column file in one single shot. but 130M 12-column file in a 2-million row chunks. For fits you can get rows and columns before hand and make a guess, for csv is not straightforward. Also, it depends on memory resources as well. I'd bet 10M is a reasonable number to start with.

[kadrlica]

It seems like it would be better to have the chunksize constraint to be in in MB rather than rows (something like upload_max_mb similar to outfile_max_mb). Is there any memory safe way to get the number of rows in a csv file? If so, we could do something like (upload_max_mb/filesize)*nrows = chunksize.

[mgckind]

I'm not sure there is one very efficient without reading the file and even from the shell command wc -l is slow for very big files. You might get a guess by reading the first line to get the number of columns and possible data types and based on the size of the file estimate the number of rows or directly chunksize

[kadrlica]

I like that idea. What about reading the first ~100 (1000?) lines and guessing at the size per line. Then dividing the file into chunks based on the total file size? I really do think that chunking in units of MB rather than rows makes a lot more sense.

[kadrlica]

Any more thoughts about this? If I get time I could make a pass at writing the code...