Issue with sub-branches coming from split custom classes

[tamasgal]

The title is a bit generic ;)

Let me rewind and show how it looked the early days of UnROOT (v0.1.7), where the getindex was different:

julia> using UnROOT

julia> f = UnROOT.ROOTFile("test/samples/km3net_online.root")
ROOTFile("test/samples/km3net_online.root") with 10 entries and 54 streamers.

julia> f["KM3NET_EVENT/KM3NET_EVENT"].fClassName
"KM3NETDAQ::JDAQEvent"

julia> keys(f["KM3NET_EVENT/KM3NET_EVENT"])
4-element Vector{String}:
 "KM3NETDAQ::JDAQPreamble"
 "KM3NETDAQ::JDAQEventHeader"
 "triggeredHits"
 "snapshotHits"

julia> f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"]
UnROOT.TBranchElement_10
  cursor: UnROOT.Cursor
  fName: String "snapshotHits"
  fTitle: String "snapshotHits"
  fFillColor: Int16 0
  fFillStyle: Int16 1001
...
...
...

Here you can see that keys(f["KM3NET_EVENT/KM3NET_EVENT"]) lists the "branches", which are fields of the KM3NETDAQ::JDAQEvent class entry.

Of course, there is some complicated way to read this class by deserialising all the four sub-branches but it's a tough thing and I haven't even managed to do so in uproot. However, It's possible to do it branch-by-branch.

With the new API we currently work on, there is currently no easy way to access subranches of such a custom branch like f["KM3NET_EVENT/KM3NET_EVENT"]:

julia> keys(f["KM3NET_EVENT/KM3NET_EVENT"])
┌ Warning: Can't automatically create LazyBranch for branch KM3NET_EVENT/KM3NET_EVENT. Returning a branch object
└ @ UnROOT ~/Dev/UnROOT.jl/src/root.jl:93
ERROR: MethodError: no method matching keys(::UnROOT.TBranchElement_10)
Closest candidates are:
  keys(::Union{Tables.AbstractColumns, Tables.AbstractRow}) at /Users/tamasgal/.julia/packages/Tables/gg6Id/src/Tables.jl:181
  keys(::Missings.EachFailMissing) at /Users/tamasgal/.julia/packages/Missings/sx5js/src/Missings.jl:154
  keys(::Dictionaries.Dictionary) at /Users/tamasgal/.julia/packages/Dictionaries/YpAxR/src/Dictionary.jl:270
  ...
Stacktrace:
 [1] top-level scope
   @ REPL[3]:1

That's quite an easy fix since we have the abstract type TBranchElement which we can use to implement keys(branch::TBranchElement).

However, the API to introduce the custom parsing currently only looks at the base class and is not able to distinguish between subbranches. This means that it only looks at fClassName and if there is a match, it will use the type in customstructs:

struct KM3NETDAQHit
   dom_id::Int32
   channel_id::UInt8
   tdc::Int32
   tot::UInt8
end

function UnROOT.interped_data(rawdata, rawoffsets, ::Type{Vector{KM3NETDAQHit}}, ::Type{J}) where J<:UnROOT.JaggType
    @views map(1:length(rawoffsets)-1) do idx
        idxrange = rawoffsets[idx]+10+1 : rawoffsets[idx+1]
        UnROOT.interped_data(rawdata[idxrange], rawoffsets[idx], KM3NETDAQHit, UnROOT.Nojagg)
    end
en

function UnROOT.interped_data(rawdata, rawoffsets, T::Type{KM3NETDAQHit}, ::Type{J}) where J <: UnROOT.JaggType
    splitdata = Base.Iterators.partition(rawdata, 10)
    out = KM3NETDAQHit[]
    sizehint!(out, length(splitdata))
    for single in splitdata
        io = IOBuffer(single)
        push!(out, T(UnROOT.readtype(io, Int32), read(io, UInt8), read(io, Int32), read(io, UInt8)))
    end
    out
end

customstructs = Dict("KM3NETDAQ::JDAQEvent" => Vector{KM3NETDAQHit})

f = UnROOT.ROOTFile("../test/samples/km3net_online.root", customstructs=customstructs)

With this code, f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"] magically becomes a jagged Vector{Vector{KM3NETDAQHit}} which is nice, but it uses the same type also for f["KM3NET_EVENT/KM3NET_EVENT/triggeredHits"] and fails.

Furthermore, the printout of f["KM3NET_EVENT/KM3NET_EVENT"] is broken when using customstructs:

MethodError: no method matching -(::Nothing, ::Int64)
Closest candidates are:
  -(::T, ::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} at int.jl:86
  -(::T, ::Integer) where T<:AbstractChar at char.jl:222
  -(::LinearAlgebra.UniformScaling, ::Number) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/uniformscaling.jl:147
  ...

Stacktrace:
  [1] getindex(ba::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg}, idx::Int64)
    @ UnROOT ~/Dev/UnROOT.jl/src/iteration.jl:111
  [2] _getindex
    @ ./abstractarray.jl:1209 [inlined]
  [3] getindex
    @ ./abstractarray.jl:1170 [inlined]
  [4] isassigned(::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg}, ::Int64, ::Int64)
    @ Base ./abstractarray.jl:513
  [5] alignment(io::IOContext{IOBuffer}, X::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg}, rows::UnitRange{Int64}, cols::UnitRange{Int64}, cols_if_complete::Int64, cols_otherwise::Int64, sep::Int64)
    @ Base ./arrayshow.jl:67
  [6] _print_matrix(io::IOContext{IOBuffer}, X::AbstractVecOrMat{T} where T, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64, rowsA::UnitRange{Int64}, colsA::UnitRange{Int64})
    @ Base ./arrayshow.jl:201
  [7] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
  [8] invokelatest
    @ ./essentials.jl:706 [inlined]
  [9] print_matrix (repeats 2 times)
    @ ./arrayshow.jl:170 [inlined]
 [10] print_array
    @ ./arrayshow.jl:327 [inlined]
 [11] show(io::IOContext{IOBuffer}, #unused#::MIME{Symbol("text/plain")}, X::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg})
    @ Base ./arrayshow.jl:368
 [12] limitstringmime(mime::MIME{Symbol("text/plain")}, x::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg})
    @ IJulia ~/.julia/packages/IJulia/e8kqU/src/inline.jl:43
 [13] display_mimestring
    @ ~/.julia/packages/IJulia/e8kqU/src/display.jl:71 [inlined]
 [14] display_dict(x::LazyBranch{Vector{KM3NETDAQHit}, UnROOT.Offsetjagg})
    @ IJulia ~/.julia/packages/IJulia/e8kqU/src/display.jl:102
 [15] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [16] invokelatest
    @ ./essentials.jl:706 [inlined]
 [17] execute_request(socket::ZMQ.Socket, msg::IJulia.Msg)
    @ IJulia ~/.julia/packages/IJulia/e8kqU/src/execute_request.jl:112
 [18] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [19] invokelatest
    @ ./essentials.jl:706 [inlined]
 [20] eventloop(socket::ZMQ.Socket)
    @ IJulia ~/.julia/packages/IJulia/e8kqU/src/eventloop.jl:8
 [21] (::IJulia.var"#15#18")()
    @ IJulia ./task.jl:411

EDIT:

So all in all, what we need is the possibility to provide the types for subbranches. Given the example above, it would be something like this:

customstructs = Dict(
    "KM3NETDAQ::JDAQEvent.snapshotHits" => Vector{KM3NETDAQHit},
    "KM3NETDAQ::JDAQEvent.triggeredHits" => Vector{KM3NETDAQTriggeredHit},
    "KM3NETDAQ::JDAQEvent.KM3NETDAQ::JDAQPreamble" => KM3NETDAQPreamble,
    "KM3NETDAQ::JDAQEvent.KM3NETDAQ::JDAQEventHeader" => KM3NETDAQEventHeader
)

[Moelf]

what's the recommended workaround for now? I'm trying to figure out how to work with Math::LorentzVector from ROOT

xref: scikit-hep/uproot5#51

basically if v_e_tlv is a ROOT::Math:: lorentz vector, then

julia> b["v_e_tlv"].fBranches.elements
4-element Vector{Any}:
 UnROOT.TBranchElement_10
  cursor: UnROOT.Cursor
  fName: String "v_e_tlv.fCoordinates.fPt"
  fTitle: String "fPt[v_e_tlv_]"

will contain the sub-branches. I guess we can make a custom interpretation interface for this too.

[tamasgal]

Yeah, we need to start working on the custom interpretation stuff, but I fear (as always) that I can't spend much time on it 😢