Lag in insert mode using large libraries

[roblevy]

As suggested here I'm making a new issue, although it appears similar to this.

I use the data analysis library pandas and when in insert mode within the parentheses of almost any pandas function, there's a very noticeable lag between responses to keystrokes.

To see this (I'm using jedi 0.9.0 and the latest jedi-vim):

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.random([3, 6]))
df.head(5).to_latex(header=False, index=True)

When the cursor is inside the parentheses associated with to_latex, the whole thing gets very laggy on my system.

Doing a profile from within vim itself reveals:

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
   24   0.012265             <SNR>19_Highlight_Matching_Pair()
   70   0.002380             jedi#_vim_exceptions()

[roblevy]

The details of 19_Highlight_Matching_Pair are:

count  total (s)   self (s)
                              " Remove any previous match.
   24              0.000319   if exists('w:paren_hl_on') && w:paren_hl_on
   19              0.000105     3match none
   19              0.000093     let w:paren_hl_on = 0
   19              0.000032   endif

                              " Avoid that we remove the popup menu.
                              " Return when there are no colors (looks like the cursor jumps).
   24              0.000260   if pumvisible() || (&t_Co < 8 && !has("gui_running"))
                                return
                              endif

                              " Get the character under the cursor and check if it's in 'matchpairs'.
   24              0.000145   let c_lnum = line('.')
   24              0.000121   let c_col = col('.')
   24              0.000063   let before = 0

   24              0.000247   let c = getline(c_lnum)[c_col - 1]
   24              0.000885   let plist = split(&matchpairs, '.\zs[:,]')
   24              0.000192   let i = index(plist, c)
   24              0.000055   if i < 0
                                " not found, in Insert mode try character before the cursor
    6              0.000068     if c_col > 1 && (mode() == 'i' || mode() == 'R')
                                  let before = 1
                                  let c = getline(c_lnum)[c_col - 2]
                                  let i = index(plist, c)
                                endif
    6              0.000016     if i < 0
                                  " not found, nothing to do
    6              0.000019       return
                                endif
                              endif

                              " Figure out the arguments for searchpairpos().
   18              0.000035   if i % 2 == 0
                                let s_flags = 'nW'
                                let c2 = plist[i + 1]
                              else
   18              0.000042     let s_flags = 'nbW'
   18              0.000043     let c2 = c
   18              0.000093     let c = plist[i - 1]
   18              0.000020   endif
   18              0.000064   if c == '['
                                let c = '\['
                                let c2 = '\]'
                              endif

                              " Find the match.  When it was just before the cursor move it there for a
                              " moment.
   18              0.000032   if before > 0
                                let save_cursor = winsaveview()
                                call cursor(c_lnum, c_col - before)
                              endif

                              " When not in a string or comment ignore matches inside them.
                              " We match "escape" for special items, such as lispEscapeSpecial.
   18              0.000107   let s_skip ='synIDattr(synID(line("."), col("."), 0), "name") ' . '=~?  "string\\|character\\|singlequote\\|escape\\|comment"'
   18              0.002330   execute 'if' s_skip '| let s_skip = 0 | endif'

                              " Limit the search to lines visible in the window.
   18              0.000095   let stoplinebottom = line('w$')
   18              0.000078   let stoplinetop = line('w0')
   18              0.000037   if i % 2 == 0
                                let stopline = stoplinebottom
                              else
   18              0.000049     let stopline = stoplinetop
   18              0.000027   endif

                              " Limit the search time to 300 msec to avoid a hang on very long lines.
                              " This fails when a timeout is not supported.
   18              0.000074   if mode() == 'i' || mode() == 'R'
   17              0.000136     let timeout = exists("b:matchparen_insert_timeout") ? b:matchparen_insert_timeout : g:matchparen_insert_timeout
   17              0.000015   else
    1              0.000015     let timeout = exists("b:matchparen_timeout") ? b:matchparen_timeout : g:matchparen_timeout
    1              0.000002   endif
   18              0.000033   try
   18              0.002733     let [m_lnum, m_col] = searchpairpos(c, '', c2, s_flags, s_skip, stopline, timeout)
   18              0.000052   catch /E118/
                                " Can't use the timeout, restrict the stopline a bit more to avoid taking
                                " a long time on closed folds and long lines.
                                " The "viewable" variables give a range in which we can scroll while
                                " keeping the cursor at the same position.
                                " adjustedScrolloff accounts for very large numbers of scrolloff.
                                let adjustedScrolloff = min([&scrolloff, (line('w$') - line('w0')) / 2])
                                let bottom_viewable = min([line('$'), c_lnum + &lines - adjustedScrolloff - 2])
                                let top_viewable = max([1, c_lnum-&lines+adjustedScrolloff + 2])
                                " one of these stoplines will be adjusted below, but the current values are
                                " minimal boundaries within the current window
                                if i % 2 == 0
                                  if has("byte_offset") && has("syntax_items") && &smc > 0
                                let stopbyte = min([line2byte("$"), line2byte(".") + col(".") + &smc * 2])
                                let stopline = min([bottom_viewable, byte2line(stopbyte)])
                                  else
                                let stopline = min([bottom_viewable, c_lnum + 100])
                                  endif
                                  let stoplinebottom = stopline
                                else
                                  if has("byte_offset") && has("syntax_items") && &smc > 0
                                let stopbyte = max([1, line2byte(".") + col(".") - &smc * 2])
                                let stopline = max([top_viewable, byte2line(stopbyte)])
                                  else
                                let stopline = max([top_viewable, c_lnum - 100])
                                  endif
                                  let stoplinetop = stopline
                                endif
                                let [m_lnum, m_col] = searchpairpos(c, '', c2, s_flags, s_skip, stopline)
                              endtry

   18              0.000031   if before > 0
                                call winrestview(save_cursor)
                              endif

                              " If a match is found setup match highlighting.
   18              0.000073   if m_lnum > 0 && m_lnum >= stoplinetop && m_lnum <= stoplinebottom 
   18              0.000590     exe '3match MatchParen /\(\%' . c_lnum . 'l\%' . (c_col - before) . 'c\)\|\(\%' . m_lnum . 'l\%' . m_col . 'c\)/'
   18              0.000065     let w:paren_hl_on = 1
   18              0.000020   endif

[davidhalter]

How do you profile vim that way?

[roblevy]

I just did exactly what alicee did here:

:profile start profile.log
:profile func *
:profile file *
" At this point do slow actions
:profile pause
:noautocmd qall!

[davidhalter]

Hmm I'm not sure if Highlight_matching_pair is really the problem, because it only takes 10 ms (which is a delay that you won't notice.

[roblevy]

Are you able to reproduce the problem?

[davidhalter]

I actually haven't tried. I don't really have time to fix jedi-vim stuff at the moment anyway. So pinning down the problem to a certain thing will definitely help to solve it in the future.

IMO this has to do with #217 and is directly related to all the other issues where somebody is noting that pandas is "slow". If you want to get rid of the lag, just disable show_call_signatures. Solving this issue should probably be done by making jedi-vim async, which is a complicated job.

[davidhalter]

This phenomenon is known to me now and I think that it's not that easy to fix. The work around is basically to do it asynchronously. Still hoping to actually implement davidhalter/jedi#385. This would solve these problems as well. Until then, just wait ;)