[BREAKING] Refactor unstack

[bkamins]

Fixes #2485

[bkamins]

In the end I try to sort and reshape without sorting if it fails.
This PR adds two kwargs which change warning behaviour we had before to error or silently allowing the result (the default is to error).

I have removed CategoricalArrays.jl dependency, but there are 2 places that are problematic because of this (both are corner cases so they can be resolved later).

[bkamins]

So maybe we should do the same and have sort=false by default, so that we can simply throw an error if sorting fails when sort=true?

I will implement sort=false and not even allow sort=true. The reason is that if someone wants to sort columns or rows later it can be done easily enough as the second step. Let us discuss how you like it when I push the changes.

[bkamins]

@nalimilan - I have thought on it more and decided to recommend a different path. I propose use groupby both for rows and for columns. The overhead should be negligible, and if there is some - we should concentrate to make groupby faster. The benefit is that we have one place to maintain the complexity of refpool etc. . Also the above test is just faster with groupby:

wide = DataFrame(id = 1:6000000,
                        a  = repeat([1:3;], inner = [2000000]),
                        b  = repeat([1:2;], inner = [3000000]),
                        c  = randn(6000000),
                        d  = randn(6000000));

long = stack(wide);
@time unstack(long, :id, :variable, :value); # master: 9.839218 seconds (6.00 M allocations: 2.726 GiB, 8.00% gc time)
                                             # with groupby: 2.086908 seconds (6.00 M allocations: 1.702 GiB, 10.32% gc time)
categorical!(long, 1:4);
@time unstack(long, :id, :variable, :value); # master: 9.277382 seconds (18.00 M allocations: 3.044 GiB, 32.15% gc time)
                                             # with groupby: 6.419417 seconds (12.00 M allocations: 2.222 GiB, 6.74% gc time)

[nalimilan]

I'm all for it if that's simpler and faster than master. Maybe just check with different shapes to be sure we don't miss something.

[bkamins]

Maybe just check with different shapes to be sure we don't miss something.

Sure.

Also this change will allow in the future to easily add support for multiple key columns - something that was requested.

[bkamins]

I went down to:

julia> @time unstack(long, :id, :variable, :value);
  1.088316 seconds (183 allocations: 598.652 MiB)

when I optimized the use of gdf.groups

EDIT: I will finalize tests tomorrow and let you know when it is good to review.

[bkamins]

This should be good for a review

[nalimilan]

For some reason I hadn't realized that using groupby makes it impossible to use the order of appearance. That's annoying, in particular as it means that even if you sort rows before calling unstack the order will be lost. Though I guess it's OK for now as we can switch to a defined order later.

[bkamins]

Though I guess it's OK for now as we can switch to a defined order later.

I think we should resolve this issue in groupby in general if we want, as mostly people will think they get the order of appearance there also, which will be not the case for PooledArray.

[bkamins]

As an additional comment. By default PooledArray correctly sorts pool by the order of appearance. The only problem is when later you reorder the values:

julia> x = PooledArray([1,2,3])
3-element PooledArray{Int64,UInt8,1,Array{UInt8,1}}:
 1
 2
 3

julia> x.pool
3-element Array{Int64,1}:
 1
 2
 3

julia> x[1:3] = 3:-1:1
3:-1:1

julia> x.pool
3-element Array{Int64,1}:
 1
 2
 3

julia> groupby(DataFrame(x=x), :x)
GroupedDataFrame with 3 groups based on key: x
First Group (1 row): x = 1
│ Row │ x     │
│     │ Int64 │
├─────┼───────┤
│ 1   │ 1     │
⋮
Last Group (1 row): x = 3
│ Row │ x     │
│     │ Int64 │
├─────┼───────┤
│ 1   │ 3     │

I think this is rare.

[bkamins]

@nalimilan - I have pushed the fix that uses the order of first appearance both for rows and for columns.

[bkamins]

Thank you!