varinfo runs out of memory #42045
Comments
|
It also runs out of memory for me, but I think that's what you would get for |
|
Are you sure that you want |
|
I'm trying to debug a script that allocates 40 gb, and only one of its dependencies is mpskit. That package does not cache anything, but the packages it depends on do tend to do that (though manually inspecting those caches only shows storage of the order 100mb). If an imported package does something weird in memory, then I want to know which it is, so that I can change it or create a pull request. Then I figured I just want the total size of all objects in memory, roughly categorized per module, which lead me to
|
Sounds like you don't really want to investigate all live objects at runtime but whant to find out where those allocations are coming from, in which case the Profile stdlib (maybe combined with ProfileView.jl) is what you probably want to take a look at. If you can't run profiling code on your full script because it takes too long, run it on a smaller test case instead of your full simulation.
When doing this full-on call on which is probably already more than you bargained for. Now imagine that but with a bunch of included modules, their dependencies, their dependencies' dependencies... These kinds of questions about where allocations are coming from are usually better suited to first ask on https://discourse.julialang.org and only after a problem in julia itself has been identified should an issue be opened. |
|
The primary question is where |
|
that probably is not meant to be read like a normal Julia code. Can't help without knowing what produced that |
|
It's the output of varinfo. varinfo internally populates a list, and then directly returns it. These are the elements of the list. |
|
No, it's definitely meant to be read like julia code. That's supposed to be an import chain. Part of the problem is that |
|
I can look at it tomorrow as well. I really suspect there's something flawed going on, where it recursively iterates over modules that it already went over. I know it'll be a lot of data, and perhaps not the most optimal tool, but I don't understand why it would fail outright. I mean, I have 120gb on that system, and I cannot do "using mpskit; varinfo(...)". Going at it the profileview kind of way is also difficult, as it certainly allocates a lot in tight loops. I don't mind it using a lot of memory there, but after all is said and done, and the simulation is finnished, the remaining data should be rather small, and it isnt. |
|
It sounds like it is having trouble recursing over the module graph, and not correctly filtering out modules that are not submodules |
|
Agreed - the code looks like it only tries to check if the child isn't the same as the parent, but ignores whether the child ever occured before? I have a small lazy lazy unique iterator, feel free to use, MIT licensedstruct UniqueIterator{T,F}
itr::T
by::F
UniqueIterator(itr::T, f::F) where {T,F} = new{T,F}(itr, f)
end
function Base.iterate(itr::UniqueIterator)
t = iterate(itr.itr)
t === nothing && return nothing
el, state = t
(el, (state, [hash(itr.by(el))]))
end
function Base.iterate(itr::UniqueIterator, (state, cache))
local el, state, h
while true
t = iterate(itr.itr, state)
t === nothing && return nothing
el, state = t
h = hash(itr.by(el))
h ∉ cache && break
yield() # so we don't hog ressources in case we never return
end
push!(cache, h)
return el, (state, cache)
end
Base.IteratorEltype(::Type{UniqueIterator{T,F}}) where {T,F} = Base.IteratorEltype(T)
Base.eltype(::Type{UniqueIterator{T,F}}) where {T,F} = eltype(T)
Base.IteratorSize(::Type{UniqueIterator{T,F}}) where {T,F} = Base.SizeUnknown() # we don't know how many unique elements there are
iunique(itr; by=identity) = UniqueIterator(itr, by)
iunique(itr...; by=identity) = UniqueIterator(Iterators.flatten(itr), by) |
|
a possible fix is this: where we just alltogether do away with prep in _populate_rows this is a very quick'n dirty hacky-ish approach though |
* InteractiveUtils: recursive correctly in varinfo, et al. Fixes #42045
…2061) * InteractiveUtils: recursive correctly in varinfo, et al. Fixes JuliaLang#42045
…2061) * InteractiveUtils: recursive correctly in varinfo, et al. Fixes JuliaLang#42045
this is on julia 1.6.2. If I call
my laptop runs out of memory. Is this not supposed to work? I have an issue where julia takes and enormous amount of memory, and I want to inspect where that is being used for.
If I modify varinfo to spit out its intermediary results, I see lines appearing like
The text was updated successfully, but these errors were encountered: