Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BREAKING] remove CategoricalArrays dependency from joins #2505

Merged
merged 5 commits into from
Nov 1, 2020
Merged

[BREAKING] remove CategoricalArrays dependency from joins #2505

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d86dc4e
remove CategoricalArrays dependency from joins
bkamins Oct 31, 2020
0543b6d
do not use internal methods of PooledArrays.jl
bkamins Oct 31, 2020
2bb1b5f
Update src/abstractdataframe/join.jl
bkamins Oct 31, 2020
b3f66a1
change indicator to UInt32 internal representation
bkamins Oct 31, 2020
a8410f9
Apply suggestions from code review
bkamins Oct 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@
choose the fast path only when it is safe; this resolves inconsistencies
with what the same functions not using fast path produce
([#2357](https://github.com/JuliaData/DataFrames.jl/pull/2357))
* joins now return `PooledVector` not `CategoricalVector` in indicator column
([#2505](https://github.com/JuliaData/DataFrames.jl/pull/2505))
* `GroupKeys` now supports `in` for `GroupKey`, `Tuple`, `NamedTuple` and dictionaries
([2392](https://github.com/JuliaData/DataFrames.jl/pull/2392))
* in `describe` the specification of custom aggregation is now `function => name`;
Expand Down
18 changes: 11 additions & 7 deletions src/abstractdataframe/join.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,10 @@ function compose_joined_table(joiner::DataFrameJoiner, kind::Symbol,
# about the permutation of left data frame in rightjoin as we always
# assign 0x1 to it anyway and these rows are guaranteed to come first
# (even if they are permuted)
left_indicator = zeros(UInt8, nrow)
left_indicator[axes(all_orig_left_ixs, 1)] .= 0x1
right_indicator = zeros(UInt8, nrow)
right_indicator[axes(all_orig_right_ixs, 1)] .= 0x2
left_indicator = zeros(UInt32, nrow)
left_indicator[axes(all_orig_left_ixs, 1)] .= 1
right_indicator = zeros(UInt32, nrow)
right_indicator[axes(all_orig_right_ixs, 1)] .= 2
permute!(right_indicator, right_perm)
end

Expand Down Expand Up @@ -416,9 +416,13 @@ function _join(df1::AbstractDataFrame, df2::AbstractDataFrame;
end

if indicator !== nothing
refs = left_indicator + right_indicator
pool = CategoricalPool{String,UInt8}(["left_only", "right_only", "both"])
indicatorcol = CategoricalArray{String,1}(refs, pool)
left_indicator .+= right_indicator
pool = ["left_only", "right_only", "both"]
invpool = Dict{String, UInt32}("left_only" => 1,
"right_only" => 2,
"both" => 3)
indicatorcol = PooledArray(PooledArrays.RefArray(left_indicator),
invpool, pool)

unique_indicator = indicator
if makeunique
Expand Down