Improve scaling of to_pandas

[vnlitvinov]

There are a few issues with to_pandas poor scaling:

1. we get partitions serially, much like in Improve scaling of from_pandas #2813, which is again a problem for Dask (but could also be a problem for Ray in multi-machine setup, where it would pull in data from other nodes serially instead of pulling them all at once). Note: this is the part to be solved in Improve scaling of to_pandas; getting all objects from partitions at once #5268
2. we reconstruct the dataframe by repeatedly calling pandas.concat() which in turn does a lot of memory copying due to how its internal structure is managed.

I wasn't able to find out a good sequence of incantations for Pandas to not copy blocks around during concatenation, so I believe that to improve performance we have to manually construct underlying Pandas block structure into a set of pre-allocated blocks (as we're solving a much simpler task than regular pandas.concat() - we know that we've split the dataframe perfectly when distributing, so there should not be any conflicts in column names or indices).

[jbrockmendel]

I wasn't able to find out a good sequence of incantations for Pandas to not copy blocks around during concatenation

Can you give an example of what you're trying to do? This may be fixed in newer pandas:

arr = np.random.randn(4, 2)
arr2 = np.random.randn(4, 3)
df = pd.DataFrame(arr)
df2 = pd.DataFrame(arr2)

res = pd.concat([df, df2], axis=1)
>>> res._mgr.nblocks
2