Slow speed of cd

[pytnik89]

After initialization of zsh-autosuggestions the speed of change dir in Midnight commander(regular movements in mc) dramaticaly falls down. I use oh-my-zsh zsh version - 5.2. Is there way to solve it?

[PythonNut]

Does this happen for only cd? I've experienced similar lag on other machines, but it affects every command. This is because of the work zsh-autosuggestions does in the precmd hook.

Right now, I'm using the following to suppress it:

# Start the autosuggestion widgets
_zsh_autosuggest_start() {
    _zsh_autosuggest_check_deprecated_config
    _zsh_autosuggest_bind_widgets
    add-zsh-hook -d precmd _zsh_autosuggest_start
}

This operates on the assumption that the list of ZLE widgets will not change after the shell has been setup. If they do, then you could potentially end up without suggestions. (Although, as far as I know, nothing apocalyptic will happen.)

See if this works for you.

I'm considering filing a pull request, but managing multiple parallel PRs is a pain and #134 is still open.

[ericfreese]

@pytnik89 please try branch fixes/remove_precmd_after_init and let us know if that solves your problems with midnight commander.

[pytnik89]

@PythonNut Thx for u async fix, it seems to work as fast as many cores I have(particularly, I spend 3 sec. to enter in directory with async against 12 sec. without it) , but ... The problem is still exist.
@ericfreese I'v tried it but with no result. Let me give some details about it: I made a lot of directories with:
for n in $(seq 100); do DN="${DN}/notsolongdirnametotestnested_$n"; done; mkdir -p "./${DN}"
And when I try to enter as deep as possible in this folder structure using midnight commander it becomes painful because of time. Without autosuggestion all is perfect, so the problem definitely in this plugin. I know that it's connected with "mc" behaviour (It's good question how mc passing full path to subshell) this why it's seems unreal to get the reason of this behaviour.

[balta2ar]

This might be related to what I reported back in the #108.

[pytnik89]

@balta2ar, can u give more details about u issue with mc, may be it's the same...

[balta2ar]

@pytnik89 I had tried to reproduce the issue on a clean Ubuntu 14.04 in Vagrant but couldn't. The problem did not reappear. However my two machines (ArchLinux and Ubuntu 12.04) are suffering from that issue. Back then I disabled autosuggestions and wanted to dig into my zshrc and binary search for the culprit. Unfortunately, I didn't get to do that since. Hopefully, I'll give it another try soon...

[pytnik89]

@balta2ar can u dig into dirictory tree as deep as possible, does it become too slow?(I'm using arch)

[balta2ar]

@pytnik89 Right now I'm on Ubuntu 12.04 and yes, it's getting way too slow. I confirm that.

I've also tried the branch and it didn't help:

zsh-autosuggestions → git rev-parse HEAD
316ea3ecd6efec02de79380bed5149531b448603

[pytnik89]

@balta2ar so we have the same issue, and I have no idea how to solve it, because of reason that I mentioned above.

[balta2ar]

@pytnik89 I've just tried on ArchLinux. For some reason cd in mc even without autosuggestions is not blazingly fast, but with it it gets way, way much slower. The initial slownes may be due to a number of addons that I use, for example z. Again, I need to filter/sort out my pile of trash in zshrc and start clean...

[balta2ar]

Well, delays are accumulating... I tried two more variants:

1. Disabled zsh vcs interaction (plugin that displays current branch in the command line)
2. Disabled z.

1 didn't visually change delay. 2 is another story. When I disabled z, cd became faster. Still, autosuggestions contribute its value to the set of delays.

[Memphizzz]

I'm also experiencing a slow prompt on Windows using babun. I can just keep any key pressed for 1-2 seconds, once I release it the prompt is still catching up spitting out characters... really annoying. I have to disable the plugin until this is fixed.

[balta2ar]

There are several complaints about mc being slow with zsh on dir change, e.g.: https://www.midnight-commander.org/ticket/2072#comment:20 Unfortunately solutions that are mentioned do not help me (things like disabling $TERM=xterm256-color, removing zsh history).

I know very little of zsh and I'm not sure how to profile it. How do I root cause this issue? I can only repeat myself that even without autosuggesions cd is slow. I looked at my preexec, precmd functions:

→ echo $preexec_functions, $precmd_functions 
_zsh_highlight_preexec_hook, _z_precmd _mc_precmd

I probably need to go deeper into them.

[balta2ar]

I also have to mention that cd in mc -u (mc without subshell) is fast. cd in plain zsh gets slower as I go down the tree, but not as slow as while running mc. And the combination of mc and zsh makes it even worse.

[balta2ar]

Another experiment that I've performed is to leave minimal zshrc with enabled autosuggestions. Without mc cd is fast. With mc it is suddenly terribly slow. I wonder what could mc be doing that it slows down cd THAT much...

[balta2ar]

Some more findings from my side. I've put code like this into autosuggestions to measure execution time:

_zsh_autosuggest_strategy_default() {
    start=$(date +%s.%N)
    local prefix="$1"

    # Get the keys of the history items that match
    local -a histkeys
    histkeys=(${(k)history[(r)$prefix*]})

    # Echo the value of the first key
    echo -E "${history[$histkeys[1]]}"
    end=$(date +%s.%N)
    echo "`date +%s.%N`: _zsh_autosuggest_strategy_default: `echo \"$end-$start\" | bc`" >> /tmp/auto.log
}

And I noticed that this function is called on every keystroke. When you're typing things in console, it's not much individually:

1463691516.825945237: _zsh_autosuggest_start: .025789318
1463691519.204096476: _zsh_autosuggest_strategy_default: .005710452
1463691519.267474753: _zsh_autosuggest_strategy_default: .005493541
1463691520.299865968: _zsh_autosuggest_strategy_default: .005329630
1463691520.787252792: _zsh_autosuggest_strategy_default: .007556564
1463691522.236169777: _zsh_autosuggest_start: .024761819
1463691525.428738204: _zsh_autosuggest_strategy_default: .007001442
1463691526.563337861: _zsh_autosuggest_strategy_default: .005066626
1463691526.623061864: _zsh_autosuggest_strategy_default: .004512419

However when I entered this directory in mc:
/tmp/1/notsolongdirnametotestnested_1/notsolongdirnametotestnested_2/notsolongdirnametotestnested_3/notsolongdirnametotestnested_4/notsolongdirnametotestnested_5/notsolongdirnametotestnested_6/notsolongdirnametotestnested_7/notsolongdirnametotestnested_8/notsolongdirnametotestnested_9/notsolongdirnametotestnested_10/notsolongdirnametotestnested_11/notsolongdirnametotestnested_12/notsolongdirnametotestnested_13/notsolongdirnametotestnested_14/notsolongdirnametotestnested_15/notsolongdirnametotestnested_16/notsolongdirnametotestnested_17/notsolongdirnametotestnested_18, I noticed about 816 calls to _zsh_autosuggest_strategy_default while this path is only about 574 characters long. Anyway, I've got this number after I summed up measurements from all the calls during a single cd: 6.314942417 (seconds). It's only time for the single function. I'm sure mc adds up time on its part as well.

My current assumption is that mc sends this long path character by character... Maybe I need to dig into mc source.

[ericfreese]

My current assumption is that mc sends this long path character by character... Maybe I need to dig into mc source.

Interesting. That would make sense. If so, this issue may be similar to #141, and may be solved with something like #134 or #148.

I know very little of zsh and I'm not sure how to profile it.

You might find zsh/zprof helpful for profiling: http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fzprof-Module

Thanks for your work!

[balta2ar]

I've been reading issues related to slow pasting and this fancy zsh's bracketed-paste-magic.

Do I get it right that if mc took advantage of bracketed-paste-magic escape codes and emulated paste while sending commands to zsh over pty, we could see performance improvements? May that be true, at least in theory?

Speaking more concretely, supposing mc is sending something like this to zsh: cd some/long/path<Enter>, if it's replaced with ^[[200~ cd some/long/path^[[201~<Enter>, could it make things better? Please correct me if I'm wrong.

The solutions you have suggested are certainly worth considering, however, to my opinion, they are plugin-specific. It occurs to me that this "service" communication with zsh that mc is performing under the hood should not be handled in the same way as plain user input by typing. At least with regards to interpreting every character as a separate keystroke. What do you think?

EDIT: mc sends things like this to zsh:

"`printf "%b" '\0057tmp\00571\0057notsolongdirnametotestnested\01371\0057notsolongdirnametotestnested\01372\0057notsolongdirnametotestnested\01373\0057notsolongdirnametotestnested\01374\0057notsolongdirnametotestnested\01375\0057notsolongdirnametotestnested\01376\0057notsolongdirnametotestnested\01377\0057notsolongdirnametotestnested\01378\0057notsolongdirnametotestnested\01379\0057notsolongdirnametotestnested\013710\0057notsolongdirnametotestnested\013711\0057notsolongdirnametotestnested\013712\0057notsolongdirnametotestnested\013713\0057notsolongdirnametotestnested\013714\0057notsolongdirnametotestnested\013715\0057notsolongdirnametotestnested\013716\0057notsolongdirnametotestnested\013717'`"`

[balta2ar]

In this commit (https://github.com/balta2ar/mc/commits/f17ebb89791ce5ff64608929f62658fa04f4556d) I did a quick-and-dirty hack to check whether using zsh bracketed paste magic can help. Basically I just send $'\e[200~' and $'\e[201~' before and after long cd command. To reproduce my mc hack and try things out locally, do this:

1. Make sure you have these lines in your .zshrc:

autoload -Uz bracketed-paste-magic
zle -N bracketed-paste bracketed-paste-magic

2. Compile mc with my hack:

git clone https://github.com/balta2ar/mc
cd mc
git checkout origin/zsh-bracketed-paste-magic
./autogen.sh
./configure
make
./src/mc

@pytnik89 Would you like to give it a try?

Personally, I think I can see the difference. It's somewhat faster, however far from perfect. I need to establish a reproducible way to measure delays.

[pytnik89]

@balta2ar, of course I will send this $'\e[200~'попробую$'\e[201~', u can additionally taste zsh_async, this speeds up zsh(depending on how many cores u have).

[pytnik89]

@balta2ar, I did, It's still thoughtful...may be a little bit faster... But thx. It's better to look into src of mc to find out the solution.

[balta2ar]

I've tried this pull request: #134
Unfortunately, I can't say it fixes the issue with mc any noticeably. The huge delay is still there...

[ericfreese]

Recently merged #178 may be a helpful workaround here. Check out the develop branch for now until v0.3.3 is released

[balta2ar]

Thank you! I think I've already mentioned that somewhere else, but just to disprove my statement from 23 of May in the same thread, async really does make a difference. I think that could be a great addition to #178.

[balta2ar]

Adding more cross references here. I mentioned this issue in rupa/z, as you can see. I just wanted to show some more numbers: rupa/z#191 (comment)

The test that I described in the above mentioned issue took 4.459800 seconds when z and autosuggestions are off, and 8.59920 seconds when async branch of autosuggestions is on. Later I measured sync branch (master) and it's 1:45.61 (1 minute, 45 s, not reported in the issue).

Basically, there is almost nothing new, but I wanted to ask whether it's possible to offload more work into background, to shrink that gap between 4.459800 and 8.59920? @ericfreese @PythonNut

[ericfreese]

@balta2ar Out of curiosity, have you tried the new features/async branch?

whether it's possible to ... shrink that gap between 4.459800 and 8.59920

I'm not sure about offloading more work into the background, but I've been thinking we could probably do some more optimization. For example if you've typed "e" and the suggestion is "echo hello" and then you type a "c", there's no need to recompute the suggestion, just use the suggestion from last time. Additionally, if you've typed an "e" and there's no suggestion, when you then type a "c" there's also no need to compute another suggestion. Since there was no suggestion with "e" you're guaranteed to get no suggestion with "ec"

[balta2ar]

Out of curiosity, have you tried the new features/async branch?

@ericfreese I've just tried it and new shell at startup prints: _zsh_autosuggest_async_recreate_pty:zle:13: Bad file descriptor number for -F: _zsh_autosuggest_async_suggestion_ready. I'm using zsh 4.3.17 (x86_64-unknown-linux-gnu). I also have zsh 5.2 and it seems to work there but I haven't run the measurements yet.

there's no need to recompute the suggestion, just use the suggestion from last time

What you say totally makes sense. Moreover, Trie is a perfect candidate for such a task. Are you going to implement something similar? One thing that bothers me is the order, though. As far as I get it, more recent items from history have preference to be suggested. This somehow needs to be accounted for during the trie construction and item retrieval.

Keeping in mind the fact that history rarely gets deleted (I suppose that's true across most users), we rarely need to reconstruct the trie. Most of the time we need to track the addition of a new history item and add it to the trie. Maybe there is a way to do it efficiently. Background task that contains this trie can serve many shells at once.

[ericfreese]

Ah, yeah it currently only works with 5.2.

Moreover, Trie is a perfect candidate for such a task. Are you going to implement something similar?

I wasn't planning on implementing a trie, though I think it might be a good data structure for zsh to implement under the hood (and who knows, maybe it already does). I want to try as hard as possible to avoid managing a separate copy of zsh history since there are so many history options we would then have to support. It's easiest if we can just delegate the tasks dealing with history to zsh builtins like fc.

[balta2ar]

and who knows, maybe it already does

I've quickly grepped "trie" and "tree" and I haven't found anything related. Maybe they name it differently though.

I want to try as hard as possible to avoid managing a separate copy of zsh history
just use the suggestion from last time

Doesn't using suggestion from the last time imply that you have some kind of a local copy of the history? It might be reduced though.

I'm not sure what's the bottleneck is at the moment, but I pictured the algorithm as follows. When a user types the first character, you retrieve the whole (or part of, depending on the configuration) history. Then you construct a data structure that allows for fast filtering. Whenever user types/deletes a character, you query that data structure to obtain new suggestion. We're storing a copy of a history only for the purpose of faster filtering and only while user in typing something in. We do not store it across different command line executions/shells (yes, this is contrast with what I described before because I think it's simpler and may be enough in terms of performance. Just another approach).

I'm just curious how you see this and how you're gonna implement it.

[attilastrba]

Hi Guys,
I am having this issue with mc that it is beeing slow with autosuggest enabled. If I understand correctly settin ZSH_AUTOSUGGEST_BUFFER_MAX_SIZE=20 should improove the situation correct? Unfortunately although if I set it to 20 I don't see any improovement.
Can you point me to the right direction what am I doing wrong here?
Thx

[ralberts]

@Memphizzz Did you find a solution that works to speed this up when using babun on windows?

[Memphizzz]

@ralberts Not really, no. I switched to fish which has auto completion by default.

[ziggythehamster]

FYI: Babun on Windows is basically just mintty running Cygwin's zsh. I'm getting this while running zsh under ConEmu (under the Cygwin/msys connector so I get proper ANSI support).

Since this build of zsh uses Cygwin, fork() is super slow, and zsh-autosuggestions is super slow. I'm trying to find something to tune, but I think there isn't anything and it's going to be super slow unless someone refactors this to use less forks.

[ericfreese]

I'm hoping that the asynchronous suggestions just merged into the develop branch will solve this issue. Please try that branch and let me know if it makes mc more usable.

[balta2ar]

Thank you very much for your work, @ericfreese! I've just checked out develop (4321fc0 (HEAD, origin/develop) We need to bind on every precmd to ensure we wrap...) branch and walked around the folders in mc. Unfortunately, on nested paths delay is still uncomfortably noticeable. At this moment, I'm not sure it can be fully solved by only making suggestions async. Maybe this is an example where efforts on many directions is required: suggestions performance, z performance, mc should use bracketed paste. Anyway, async feature has definitely mitigated the original issue with mc, thank you for that. I'm only noticing that in my set up there is something else that contributes to delays.

Besides, when I bring console in mc by pressing Ctrl-O and run any command after traveling across directories for some time, I get this printed after the command output:

/home/bz
_zsh_autosuggest_async_pty_create:zpty:12: pty command name already used: zsh_autosuggest_pty    
_zsh_autosuggest_async_pty_create:zle:22: Bad file descriptor number for -F: _zsh_autosuggest_async_response
_zsh_autosuggest_async_pty_create:zpty:12: pty command name already used: zsh_autosuggest_pty
_zsh_autosuggest_async_pty_create:zle:22: Bad file descriptor number for -F: _zsh_autosuggest_async_response

EDIT: On top of that I notice high zsh's CPU usage.

[ericfreese]

Thanks @balta2ar 😄 I will look into those error messages tomorrow.

[ericfreese]

Ok @balta2ar, I've pushed up a few commits that I believe fix the issue causing those error messages. Please give it another shot when you have time. Thanks again for your patience and taking the time to try this new stuff out. It's super helpful 🍻

[balta2ar]

Wow, not only errors are gone, but I also feel that delay on directory change in mc is noticeably smaller! I'm not saying it's perfect, I can still create a rather artificial example of highly nested directories (mentioned here #136 (comment)) and it starts to get slower there, but it's definitely better for my practical day-to-day use cases (comparing to what I saw in the morning today). Thank you for your efforts, @ericfreese! I appreciate it very much!

On a side note, and out of curiosity, do you still plan to have optimizations you mentioned above: #136 (comment)? I think having them along with async would be amazing.

[ericfreese]

Awesome!

On a side note, and out of curiosity, do you still plan to have optimizations you mentioned above: #136 (comment)? I think having them along with async would be amazing.

I've implemented the first half of that on develop here:

if you've typed "e" and the suggestion is "echo hello" and then you type a "c", there's no need to recompute the suggestion, just use the suggestion from last time.

I think we do still try to fetch a suggestion on every keystroke if there was no prior suggestion though, so that could probably be optimized a bit further. I'll try to get that in for v0.4.0

[balta2ar]

Just noticed: when I type - (minus sign), it prints this:

→ -            
_zsh_autosuggest_async_response:zle:6: number expected after -n

[ericfreese]

Ok, should be fixed with c9a51e0. Thanks!