exa --tree --all --git-ignore is extremely slow

[ahmedelgabri]

• exa --tree --all works but slow
• exa --tree --git-ignore works & fast
• exa --tree --all --git-ignore I had to kill it after 30sec.

I ran this on my dotfiles, but keep in mind that some folders contain lots of generated files on desk that are not commited or being part of the repo. That's why I wanted to try --git-ignore

[t-8ch]

@ahmedelgabri Does the second invocation actually respect .gitignore?
It does not for me.

[ahmedelgabri]

@ahmedelgabri Does the seconds invocation actually respect .gitignore?
It does not for me.

Just checked now & yes, it doesn't respect .gitignore

[vmchale]

Relatedly, a plain exa -T is slow. Running it on the ghc repo, I find

benchmarking bench/exa -T
time                 700.6 ms   (691.7 ms .. 707.7 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 701.6 ms   (700.4 ms .. 703.5 ms)
std dev              1.829 ms   (448.8 μs .. 2.422 ms)
variance introduced by outliers: 19% (moderately inflated)

benchmarking bench/tree -C
time                 148.2 ms   (147.8 ms .. 148.5 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 148.2 ms   (148.0 ms .. 148.4 ms)
std dev              309.8 μs   (211.8 μs .. 408.9 μs)
variance introduced by outliers: 12% (moderately inflated)

I suppose there is some room for improvement.

[tekumara]

exa --tree --git-ignore is extremely slow (7 mins vs 700ms) for me too:

$ time exa --tree > /dev/null 
exa --tree > /dev/null  0.62s user 1.38s system 264% cpu 0.752 total

$ time exa --tree --git-ignore > /dev/null
exa --tree --git-ignore > /dev/null  426.81s user 4.51s system 100% cpu 7:11.31 total

[imajes]

issue #265 also related. @ogham @ariasuni -- any chance this could get looked at? I'd love to know why these get so slow compared to normal -- and could they be timed out more sensibly?

[ariasuni]

exa --tree --git-ignore seems to be really slow with the current stable version, but using --git-ignore with version built from source is actually faster on a repository with a lot of files to ignore.

I tested with tootsuite/mastodon, I tried with current stable version and stopped it after waiting for 3m30, but now:

❯ time exa --tree > /dev/null

________________________________________________________
Executed in    1,97 secs   fish           external 
   usr time    1,19 secs  485,00 micros    1,19 secs 
   sys time    3,18 secs    0,00 micros    3,18 secs 

❯ time exa --tree --git-ignore > /dev/null

________________________________________________________
Executed in  245,73 millis    fish           external 
   usr time  153,80 millis    0,00 micros  153,80 millis 
   sys time  187,69 millis  638,00 micros  187,05 millis

❯ time exa --tree --all --git-ignore ~/projets/contrib/mastodon > /dev/null

________________________________________________________
Executed in  289,29 millis    fish           external 
   usr time  169,47 millis    0,00 micros  169,47 millis 
   sys time  270,63 millis  1222,00 micros  269,41 millis

@ahmedelgabri do you think can build exa from source and tell us if it’s fixed for you?

[ahmedelgabri]

These are my results testing against the same repo as @ariasuni (tootsuite/mastodon), I use nix to install exa which already builds from source.

It's much much better than before for sure.

tmp/foo/mastodon
» exa -v
exa v0.9.0

tmp/foo/mastodon
» time exa --tree > /dev/null
exa --tree > /dev/null  0.09s user 0.15s system 132% cpu 0.188 total

tmp/foo/mastodon
» time exa --tree --git-ignore > /dev/null
exa --tree --git-ignore > /dev/null  3.38s user 0.21s system 103% cpu 3.459 total

tmp/foo/mastodon
» time exa --tree --git-ignore --all > /dev/null
exa --tree --git-ignore --all > /dev/null  3.70s user 0.22s system 102% cpu 3.817 total

[cho-m]

Are any of the original performance issues discussed in #234 (comment) resolved?
Quoted below:

... In this instance, I'm aware of two major slowdowns that would affect tree view, and coincidentally they have reversible issue numbers:

• Consider buffered algorithm to handle recursion more gracefully #82 deals with the fact that exa will buffer the entire tree view before displaying even the first line. This is a holdover from the days when you had to display the --long table to have a tree at all, which did require buffering to wrap correctly, but that's not the case anymore. While this doesn't affect the output (I've run exa --tree / before, and it printed it all out eventually), it still requires more heap allocation and re-allocation that tree doesn't need to do at all.
• libgit2 git_status_list_new is really slow. #28 is about Git support. 99% of the time someone says that exa runs slowly, the slow Git implementation is the culprit. I know you mentioned it's slow indeterminate of the --git flag (and I believe you! In fact I'm surprised you didn't say it made it even slower) but I'm mentioning it here for others.
• There's undoubtably also many small optimisations that could be made here and there: unnecessary heap allocations that could be elided, redundant checking, etc.

---

Currently, for simply showing large trees, I use fd via:

• fd . (use gitignore)
• fd -I . (disable gitignore)
• fd -H . (hidden files, use gitignore)
• fd -HI . (hidden files, disabled gitignore)

and just pipe it into either tree --fromfile . or as-tree. I usually find this runs faster (benchmark at bottom. output via wc -l shows similar data with exa lacking the last 2 lines for # dirs/files in tree).

Of course, this lacks support for extra details in --long and aesthetic enhancements like --icons, so tree-rendering performance improvements in exa would be nice.

% hyperfine --warmup 3 'tree | wc -l' 'exa --tree | wc -l' 'fd -I . | tree --fromfile .  | wc -l'
Benchmark #1: tree | wc -l
  Time (mean ± σ):      4.895 s ±  0.157 s    [User: 1.600 s, System: 3.207 s]
  Range (min … max):    4.727 s …  5.150 s    10 runs

Benchmark #2: exa --tree | wc -l
  Time (mean ± σ):      5.857 s ±  0.086 s    [User: 1.931 s, System: 6.765 s]
  Range (min … max):    5.766 s …  6.036 s    10 runs

Benchmark #3: fd -I . | tree --fromfile .  | wc -l
  Time (mean ± σ):      1.000 s ±  0.018 s    [User: 1.718 s, System: 1.784 s]
  Range (min … max):    0.965 s …  1.028 s    10 runs

Summary
  'fd -I . | tree --fromfile .  | wc -l' ran
    4.89 ± 0.18 times faster than 'tree | wc -l'
    5.86 ± 0.14 times faster than 'exa --tree | wc -l'

% tree | wc -l
  380720
% fd -I . | tree --fromfile . | wc -l
  380720
% exa --tree | wc -l
  380718

[ariasuni]

@ahmedelgabri When I wrote «build exa from source», I meant «build exa from the latest source». Anyway, now that 0.10.0 is released, could you try and tell me if it fixed the problem for you?

[zrma]

@ahmedelgabri When I wrote «build exa from source», I meant «build exa from the latest source». Anyway, now that 0.10.0 is released, could you try and tell me if it fixed the problem for you?

Dear @ahmedelgabri

in my case, before prev 0.10.0 it works fast, --git-ignore option, but nowadays after 0.10.1 it works extremely slow.

$ exa -v
exa - list files on the command-line
v0.10.1 [+git]
https://the.exa.website/

$ time exa --git-ignore

real    0m26.812s
user    0m0.778s
sys     0m4.978s

$ time exa

real    0m0.004s
user    0m0.002s
sys     0m0.000s

thank you.

[ahmedelgabri]

I ran the same exact test I ran earlier #385 (comment) & for me, there is a big & clear improvement between 0.9.0 & 0.10.1

Worth mentioning that I'm testing this on a 6 years old machine

MacBook Pro (Retina, 15-inch, Mid 2015)
2,5 GHz Quad-Core Intel Core i7 with 16GB of RAM

So I expect it might be even faster on new hardware.

tmp/foo/mastodon
» exa -v
exa - list files on the command-line
v0.10.1 [+git]
https://the.exa.website/

tmp/foo/mastodon
» time exa --tree > /dev/null
exa --tree > /dev/null  0.10s user 0.16s system 129% cpu 0.196 total

tmp/foo/mastodon
» time exa --tree --git-ignore > /dev/null
exa --tree --git-ignore > /dev/null  0.17s user 0.45s system 121% cpu 0.515 total

tmp/foo/mastodon
» time exa --tree --git-ignore --all > /dev/null
exa --tree --git-ignore --all > /dev/null  0.20s user 0.53s system 112% cpu 0.642 total

So I'm closing this issue as it already fixed the issue for me. And thanks to everyone for the help!