show_func_def() can be slow

[mattjj]

It may have to do with larger modules or modules with many dependencies causing a lot of reads. An example is a file like

import scipy.weave
scipy.weave.inline(

When the function definition is pulled up after typing that, not only is it slow to load the definition right when the open-paren is typed, but also typing each character afterwards (in the argument list) has significant lag. The issue may be solvable with caching (e.g. caching python's standard libraries into jedi-vim's internal data structures so that they don't need to be re-parsed, if that's what's happening, as well as caching whatever computation is happening when each character is typed).

It may be useful to use the profilehooks module like this in plugin/jedi_vim.py:

import sys
sys.stdout = open('/Users/mattjj/Desktop/blah','w')
from profilehooks import profile
@profile(immediate=True)
def show_func_def(call_def=None, completion_lines=0):
    ...

It looked like much of the time was spent in file I/O.

This issue may be related to #20, but it comes up when the source file being edited is very small (as above).

[davidhalter]

Yes, I know about that. But thanks for posting anyway, because I really need good examples. It's a top priority.

[davidhalter]

Hey I tried it now with a newer version of jedi (but actually that shouldn't make a big difference...). Could you please pull and try again? Because I don't feel any slowness.

Btw: I know that loading the scipy library for the first time is/might be slow. I don't think I can ever fix that, because some libraries like scipy/numpy are really big (star imports and alike). But after the first time using it, it should be fast (because of caching).

[mattjj]

It does seem much better with a fresh pull of jedi, but typing is still slow for certain functions (like weave), and that's after the initial library load. Maybe the python side is now fast and the slowness I'm noticing is in vim; this weave function has a very long list of arguments, so it may be that re-displaying that long argument list with every character typed (i.e. calls to vim's conceal) is producing the lag I'm noticing.

Did you try it with my example and not notice any lag?

[davidhalter]

No, I didn't notice anything. How fast is your computer?

Currently my main priority is speeding up many things, so I guess there will be a few improvements next week.

[mattjj]

I think it's pretty fast: it's a mid-2010 MacBook Pro with a 2.4 GHz i5 and an OCZ Vertex 3 SSD (which is where the libraries and edited files live). The lag is maybe only around 0.1-0.2 seconds per character, but it's noticeable when typing. I'll look into it more and see if I can figure something out, especially since your setup may not be producing the same effect.

[davidhalter]

Hmm yeah, now I know which lag you mean. The lag before in #20 was just much worse, so I didn't notice it.

I'm working on it.

[mattjj]

No rush; overall it's functioning really well for me! I singled out that weave example because it was the only place I noticed some lag; other function definitions pop up fast and are very helpful.

[davidhalter]

haha. I found it. It's checking numpy every single time! This means ~300 file checks :-)

I just have to find a fix... This might be more difficult ;-) To many things depend on the import mechanism...

Edit: I realized that this has to wait. I want to fix it right (which means also fast numpy lookups).
If you want a fast function, just use from scipy import weave; weave.inline. The architecture of Jedi has problems with numpy anways, so I want to find faster solutions there.

[ugtar]

It seems to also be slow if you use it with a massive file open. I have a ~5.1k line python file (I know, I know.. but it's out of my control) and with show_func_def enabled I can barely type. I don't think that in my case it's the imports because I tried creating a new file and importing all the same modules and its still snappy. Am I SOL on this one?

Edit: oops, looks like you already have issue #20 for what I'm seeing...

[davidhalter]

davidhalter/jedi@586dcc2 I probably fixed this (just the scipy.weave` part...)

@mattjj: Could you please test it once? (The fix is in the dev branch, not within master).

@ugtar: I improved many performance things, could you tell me how bad it still is? (also, fixes are in dev branch)

[mattjj]

It's still a bit laggy for me while typing, though it might be better. When this sort of thing is likely to come up, I leave show_func_def off.

[davidhalter]

Does it feel laggy all the times? Or is it just a problem within scipy.weave.inline? Because I really have a hard time to feel the lag :-) It's almost gone I would say.

[davidhalter]

Oh, what I forgot to mention: I didn't talk about the jedi-vim dev branch, I've been talking about the jedi dev branch.

[mattjj]

Oops, you're right, I pulled the wrong thing.

And the lag is pretty much imperceptible now! Wow, nice work!

Thanks for guessing and correcting my error.

[davidhalter]

BTW: I pushed it all to master now (because it was working out). There are quite a few changes, which makes Jedi perform better. Just use the master branch again!

@ugtar Although I'm closing this, I'm still very interested in comments on "real-life" experience in the work with big files! But actually the better place to discuss this is #20.

[ugtar]

Sure thing! I'll let you know how it looks as soon as I can test it out.
Thanks

Uri

Please consider the environment before printing this message.
http://wwf.panda.org/savepaper/

On Nov 21, 2012, at 6:57 PM, David Halter notifications@github.com wrote:

BTW: I pushed it all to master now (because it was working out). There are
quite a few changes, which makes Jedi perform better. Just use the master
branch again!

@ugtar https://github.com/ugtar Although I'm closing this, I'm still
interested in comments on "real-life" experience in the work with big files!

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/23#issuecomment-10605117.

[ugtar]

Amazing improvement! I'm typing in this 6000 line file with show_function_definition on and there is no perceptible lag while typing! Popping up the function arguments when I open parens still incurs a slight lag, but really not a big deal. However, I still see a couple of issues...

First, the function popup doesn't always show up. It seems like this happens when I start typing a class method in another method of the class above the return. For example, when I type self.testa( in this snippet, the popup doesn't appear:

class Test(object):
        def testa(self, a, b):
                pass

        def testb(self, b):
                self.testa(
                return b

But, if I type self.testa( after the return line, or remove the return line, the popup appears again. This happens in a file of any size.

Second, it is still sometimes very laggy while I type a method's arguments, but it gets quick again once I close the method's parens. Also, this is not true all the time. I can't really tell what the difference is between when it's slow or fast.

Incidentally, it seems you fixed the other problem I was seeing with the function definition popup (when the popup does appear) (issue #19/#21).

[davidhalter]

I'm trying to improve the performance further. But It'll take a while :-) Basically the parser is the problem (because the parser processes everything) and the fixes are just a little workaround to reduce this. The idea is now to make a parser that processes only part of a file. But that's really tricky and might need some time.

I probably add some debugging functions to measure speed (I'm not sure what's the bottleneck in some cases).