Improve over-estimating for ORC coalescing reading

[wbo4958]

This PR is for issue #2945 .

Let's assume we're coalescing 2 orc files with 2 stripes for each file in a Spark Task.

The original initial estimated size like below,

size_of (“ORC”) + 
size_of stripe1 (DATA + StripeFooter) +
size_of stripe2 (DATA + StripeFooter) + 
size_of stripe3 (DATA + StripeFooter) + 
size_of stripe4 (DATA + StripeFooter) +
size_of FOOTER1 +
size_of FOOTER2  +
256 (Max size of PostScript) + 
1 (Length of PostScript) +
128 * 1024 (Add in a bit of fudging in case the whole file is being consumed and
                   our codec version isn't as efficient as the original writer's codec.)

We have added all ORC file's FOOTER size (size_of FOOTER1 + size_of FOOTER2) in order to estimate the StripeInformation size in FOOTER. It seems a little over-estimating since FOOTER size includes not only StripeInformation but also Schema/ColumnStatistics ...

This PR uses a fixed worst-case StripeInformation size instead of using ORC files' FOOTER size.

Below is the comparing test on 5000 orc files total 1.3G

over-estimating = initialized estimated size - the final true size

spark task number	upstream over-estimating size	with this PR over-estimating size
12	14.1M	1.6M
162	32.7M	20.6M

Please be note, we have added 128 * 1024 as the fudging. If we reduce this size, the over-estimating can only be 1.6M - 12 * (128*1024) = 102K and 20.6M - 162 * (128*1024) = 358.4K

[wbo4958]

build

[pxLi]

build

[wbo4958]

@tgravescs Hi Tom, Could you help to review this PR?

[tgravescs]

what is the logic behind the 128 * 1024 size? Have we run into cases we actually hit this and what was it?

[tgravescs]

overall looks fine, just curious if we can improve or document on the 128*1024.

[wbo4958]

overall looks fine, just curious if we can improve or document on the 128*1024.

@tgravescs I don't know, it's from the very beginning. From the comments, looks like the different codec between the original file and the current one we use may consume different buffer.