Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "Generalize broadcast!(f, ::BitVector) optimization to BitArray." #52775

Merged
merged 1 commit into from
Jan 6, 2024
Merged

Revert "Generalize broadcast!(f, ::BitVector) optimization to BitArray." #52775

merged 1 commit into from
Jan 6, 2024

Conversation

maleadt
Copy link
Member

@maleadt maleadt commented Jan 6, 2024

Reverts #52736; cc @N5N3. This broke many packages on PkgEval.

Error 1: MultiScaleTreeGraph.jl

  Got exception outside of a @test
  MethodError: no method matching SubDataFrame(::DataFrame, ::CartesianIndex{1}, ::Colon)
  
  Closest candidates are:
    SubDataFrame(::DataFrame, !Matched::Colon, ::Any)
     @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:86
    SubDataFrame(::DataFrame, !Matched::AbstractVector{Bool}, ::Any)
     @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:98
    SubDataFrame(::DataFrame, !Matched::AbstractVector{Int64}, ::Any)
     @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:80
    ...
  
  Stacktrace:
    [1] copyto!
      @ ./broadcast.jl:0 [inlined]
    [2] copyto!
      @ Base.Broadcast ./broadcast.jl:924 [inlined]
    [3] copy
      @ Base.Broadcast ./broadcast.jl:896 [inlined]
    [4] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, DataFrames.var"#103#104"{MultiScaleTreeGraph.var"#130#132"{String}}, Tuple{DataFrames.DataFrameRows{DataFrame}}})
      @ Base.Broadcast ./broadcast.jl:871
    [5] _filter_helper(f::Function, cols::DataFrames.DataFrameRows{DataFrame})
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1216
    [6] #filter#92
      @ ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1189 [inlined]
    [7] filter
      @ ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1188 [inlined]
    [8] paste_node_mtg(mtg::Node{MutableNodeMTG, Dict{Symbol, Any}}, features::DataFrame)
      @ MultiScaleTreeGraph ~/.julia/packages/MultiScaleTreeGraph/gizDZ/src/write_mtg/write_mtg.jl:131
    [9] (::MultiScaleTreeGraph.var"#126#128"{Node{MutableNodeMTG, Dict{Symbol, Any}}, DataFrame, Nothing, DataFrame})(io::IOStream)
      @ MultiScaleTreeGraph ~/.julia/packages/MultiScaleTreeGraph/gizDZ/src/write_mtg/write_mtg.jl:92
   [10] open(::MultiScaleTreeGraph.var"#126#128"{Node{MutableNodeMTG, Dict{Symbol, Any}}, DataFrame, Nothing, DataFrame}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
      @ Base ./io.jl:408
   [11] open
      @ ./io.jl:405 [inlined]
   [12] write_mtg(file::String, mtg::Node{MutableNodeMTG, Dict{Symbol, Any}}, classes::DataFrame, description::Nothing, features::DataFrame)
      @ MultiScaleTreeGraph ~/.julia/packages/MultiScaleTreeGraph/gizDZ/src/write_mtg/write_mtg.jl:49
   [13] #write_mtg#125
      @ MultiScaleTreeGraph ~/.julia/packages/MultiScaleTreeGraph/gizDZ/src/write_mtg/write_mtg.jl:44 [inlined]
   [14] write_mtg(file::String, mtg::Node{MutableNodeMTG, Dict{Symbol, Any}})
      @ MultiScaleTreeGraph ~/.julia/packages/MultiScaleTreeGraph/gizDZ/src/write_mtg/write_mtg.jl:29
   [15] (::var"#55#56")(f::String, io::IOStream)
      @ Main ~/.julia/packages/MultiScaleTreeGraph/gizDZ/test/test-write_mtg.jl:9
   [16] mktemp(fn::var"#55#56", parent::String)
      @ Base.Filesystem ./file.jl:771
   [17] mktemp(fn::Function)
      @ Base.Filesystem ./file.jl:769
   [18] macro expansion
      @ ~/.julia/packages/MultiScaleTreeGraph/gizDZ/test/test-write_mtg.jl:8 [inlined]
   [19] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
   [20] top-level scope
      @ ~/.julia/packages/MultiScaleTreeGraph/gizDZ/test/test-write_mtg.jl:8
   [21] include(fname::String)
      @ Main ./sysimg.jl:38
   [22] macro expansion
      @ ~/.julia/packages/MultiScaleTreeGraph/gizDZ/test/runtests.jl:38 [inlined]
   [23] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
   [24] top-level scope
      @ ~/.julia/packages/MultiScaleTreeGraph/gizDZ/test/runtests.jl:38
   [25] include(fname::String)
      @ Main ./sysimg.jl:38
   [26] top-level scope
      @ none:6
   [27] eval
      @ Core ./boot.jl:428 [inlined]
   [28] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:291
   [29] _start()
      @ Base ./client.jl:525

This error also seen with CrystalInfoFramework, Extremes, DTables, ...

Error 2: ReadStatTables.jl

  MethodError: no method matching LabeledArray(::Int64, ::Dict{Union{Missing, Int64}, String})
  
  Closest candidates are:
    LabeledArray(!Matched::AbstractArray{V, N}, ::Dict{K, String}) where {V, N, K}
     @ ReadStatTables ~/.julia/packages/ReadStatTables/jDyWh/src/LabeledArrays.jl:219
  
  Stacktrace:
    [1] getindex
      @ ReadStatTables ~/.julia/packages/ReadStatTables/jDyWh/src/LabeledArrays.jl:326 [inlined]
    [2] _broadcast_getindex
      @ Base.Broadcast ./broadcast.jl:643 [inlined]
    [3] _getindex
      @ Base.Broadcast ./broadcast.jl:673 [inlined]
    [4] _broadcast_getindex
      @ Base.Broadcast ./broadcast.jl:649 [inlined]
    [5] getindex
      @ Base.Broadcast ./broadcast.jl:604 [inlined]
    [6] getindex
      @ Base.Broadcast ./broadcast.jl:611 [inlined]
    [7] macro expansion
      @ Base.Broadcast ./broadcast.jl:1007 [inlined]
    [8] macro expansion
      @ Base.Broadcast ./simdloop.jl:77 [inlined]
    [9] copyto!
      @ Base.Broadcast ./broadcast.jl:1006 [inlined]
   [10] copyto!
      @ Base.Broadcast ./broadcast.jl:924 [inlined]
   [11] copy
      @ Base.Broadcast ./broadcast.jl:896 [inlined]
   [12] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(==), Tuple{LabeledVector{Int64, Vector{Int64}, Union{Missing, Int64}}, Int64}})
      @ Base.Broadcast ./broadcast.jl:871
   [13] macro expansion
      @ ~/.julia/packages/ReadStatTables/jDyWh/test/LabeledArrays.jl:130 [inlined]
   [14] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.11/Test/src/Test.jl:676 [inlined]
   [15] macro expansion
      @ ~/.julia/packages/ReadStatTables/jDyWh/test/LabeledArrays.jl:130 [inlined]
   [16] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
   [17] top-level scope
      @ ~/.julia/packages/ReadStatTables/jDyWh/test/LabeledArrays.jl:56

This error also seen with ReadStatTables, ...

Error 3: DrillHoles.jl

  ArgumentError: invalid row index: 1 of type Int64
  Stacktrace:
    [1] SubDataFrame
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:88 [inlined]
    [2] SubDataFrame
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:124 [inlined]
    [3] view
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/subdataframe/subdataframe.jl:146 [inlined]
    [4] getindex
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/iteration.jl:84 [inlined]
    [5] _broadcast_getindex
      @ Base.Broadcast ./broadcast.jl:643 [inlined]
    [6] _getindex
      @ Base.Broadcast ./broadcast.jl:674 [inlined]
    [7] _broadcast_getindex
      @ Base.Broadcast ./broadcast.jl:649 [inlined]
    [8] getindex
      @ Base.Broadcast ./broadcast.jl:604 [inlined]
    [9] getindex
      @ Base.Broadcast ./broadcast.jl:611 [inlined]
   [10] macro expansion
      @ Base.Broadcast ./broadcast.jl:1007 [inlined]
   [11] macro expansion
      @ Base.Broadcast ./simdloop.jl:77 [inlined]
   [12] copyto!
      @ Base.Broadcast ./broadcast.jl:1006 [inlined]
   [13] copyto!
      @ Base.Broadcast ./broadcast.jl:924 [inlined]
   [14] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, DataFrames.var"#103#104"{DrillHoles.var"#7#8"{Float64, Float64}}, Tuple{DataFrames.DataFrameRows{SubDataFrame{DataFrame, DataFrames.Index, Vector{Int64}}}}})
      @ Base.Broadcast ./broadcast.jl:896
   [15] materialize
      @ ./broadcast.jl:871 [inlined]
   [16] _filter_helper(f::Function, cols::DataFrames.DataFrameRows{SubDataFrame{DataFrame, DataFrames.Index, Vector{Int64}}})
      @ DataFrames ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1216
   [17] #filter#92
      @ ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1189 [inlined]
   [18] filter
      @ ~/.julia/packages/DataFrames/58MUJ/src/abstractdataframe/abstractdataframe.jl:1188 [inlined]
   [19] interleave(itables::Vector{DataFrame})
      @ DrillHoles ~/.julia/packages/DrillHoles/cxUUI/src/desurvey.jl:193
   [20] desurvey(collar::Collar{DataFrame}, survey::Survey{DataFrame}, intervals::Vector{Interval{DataFrame}}; step::Symbol, indip::Symbol, outdip::Symbol, len::Nothing, geom::Symbol, radius::Float64)
      @ DrillHoles ~/.julia/packages/DrillHoles/cxUUI/src/desurvey.jl:57
   [21] macro expansion
      @ ~/.julia/packages/DrillHoles/cxUUI/test/runtests.jl:13 [inlined]
   [22] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.11/Test/src/Test.jl:1598 [inlined]
   [23] top-level scope
      @ ~/.julia/packages/DrillHoles/cxUUI/test/runtests.jl:8
   [24] include(fname::String)
      @ Main ./sysimg.jl:38
   [25] top-level scope
      @ none:6
   [26] eval
      @ Core ./boot.jl:428 [inlined]
   [27] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:291
   [28] _start()
      @ Base ./client.jl:525

@maleadt maleadt added the revert This reverts a previously merged PR. label Jan 6, 2024
@maleadt maleadt merged commit 11bb62b into master Jan 6, 2024
5 of 8 checks passed
@maleadt maleadt deleted the revert-52736-bitarray_bc branch January 6, 2024 11:43
@N5N3
Copy link
Member

N5N3 commented Jan 6, 2024

I'm affraid these are some indexing "bug" on package side.
Error 1 and 3 should be fixed by JuliaData/DataFrames.jl#3413.
Error 2 should be caused by this overloading

Base.@propagate_inbounds function Base.getindex(x::LabeledArrOrSubOrReshape{V,N},
        I::Vararg{Any,N}) where {V,N}
    val = refarray(x)[I...]
    return LabeledArray(val, getvaluelabels(x))
end

which assumes that refarray(x)[I...] returns a AbstractArray, which is of wrong for CartesianIndex.
Their broadcast has been broken for nd case for a long time

julia> LabeledArray(ones(3, 3), Dict(1. => "1")) .|> identity
ERROR: MethodError: no method matching LabeledArray(::Float64, ::Dict{Float64, String})

Closest candidates are:
  LabeledArray(::AbstractArray{V, N}, ::Dict{K, String}) where {V, N, K}
   @ ReadStatTables C:\Users\MYJ\.julia\packages\ReadStatTables\jDyWh\src\LabeledArrays.jl:219

We can restore the "old" 1d broadcast indexing behavior in Base but I'm not sure if we need a test for this.

@maleadt
Copy link
Member Author

maleadt commented Jan 6, 2024 •
edited
Loading

Can we land (and tag) those before merging this PR again? Otherwise it's very hard to find actual regressions in the PkgEval output.

N5N3 added a commit to N5N3/julia that referenced this pull request Jan 6, 2024
@N5N3
Revert "Revert "Generalize broadcast!(f, ::BitVector) optimization …
ae6af52 
…to `BitArray`." (JuliaLang#52775)"

This reverts commit 11bb62b.
@N5N3 N5N3 mentioned this pull request Jan 6, 2024
Merged
@N5N3
Copy link
Member

N5N3 commented Jan 6, 2024

@maleadt #52776 should make PkgEval happy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
revert This reverts a previously merged PR.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maleadt @N5N3