You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
there's a super free optimization that is especially effectively for network I/O:
First:
diff --git a/src/iteration.jl b/src/iteration.jl
index 04e5d7d..7e30835 100644
--- a/src/iteration.jl+++ b/src/iteration.jl@@ -333,7 +333,7 @@ function Base.getindex(ba::LazyBranch{T,J,B}, range::UnitRange) where {T,J,B}
ib2 = findfirst(x -> x > (last(range) - 1), ba.fEntry) - 1
offset = ba.fEntry[ib1]
range = (first(range)-offset):(last(range)-offset)
- return vcat([basketarray(ba, i) for i in ib1:ib2]...)[range]+ return Vcat(asyncmap(i->basketarray(ba, i), ib1:ib2)...)[range]
end
but to make this useful, we also need:
diff --git a/src/root.jl b/src/root.jl
index e4eb852..abb1c09 100644
--- a/src/root.jl+++ b/src/root.jl@@ -472,17 +472,17 @@ function readbasket(f::ROOTFile, branch, ith)
end
function readbasketseek(f::ROOTFile, branch::Union{TBranch, TBranchElement}, seek_pos::Int, nb)
- lock(f)- local basketkey, compressedbytes+ # lock(f)+ local rawbuffer
try
seek(f.fobj, seek_pos)
rawbuffer = OffsetBuffer(IOBuffer(read(f.fobj, nb)), seek_pos)
- basketkey = unpack(rawbuffer, TBasketKey)- compressedbytes = compressed_datastream(rawbuffer, basketkey)
catch
finally
- unlock(f)+ # unlock(f)
end
+ basketkey = unpack(rawbuffer, TBasketKey)+ compressedbytes = compressed_datastream(rawbuffer, basketkey)
Now, the problem of course is that removing the lock will fuck up our multi-threaded reading, because the current cursor location is stored in IOStream. Technically, network based ones don't need a position() since each read is bytes range based, but our on-disk IOStream will surely complain.
One possibility is to use MMap based for on-disk files
or we can "duplicate" cursor object for each file for each thread...
The text was updated successfully, but these errors were encountered:
there's a super free optimization that is especially effectively for network I/O:
First:
but to make this useful, we also need:
Now, the problem of course is that removing the
lock
will fuck up our multi-threaded reading, because the current cursor location is stored in IOStream. Technically, network based ones don't need aposition()
since each read is bytes range based, but our on-disk IOStream will surely complain.MMap
based for on-disk filesThe text was updated successfully, but these errors were encountered: