High CPU usage when piping into less? #258
Comments
|
That's... strange. I kind of wonder whether it's related to #200, but I don't think it is. I can't actually reproduce this either. If I run What version of ripgrep are you running? OS? Shell? |
BurntSushi
added
the
question
|
I'm on macOS 10.12.1, using ripgrep 0.3.1 and fish, version 2.4.0. I can also reproduce this in bash 3.2.57. |
|
@danielcompton Can you try this? (So we're searching the exact same data.) |
|
Interestingly, the behaviour doesn't show up on your linux example. I was able to create a minimal example at https://gist.github.com/danielcompton/cc667b4904609de88c9008f60e11564c. Having those two files in a directory is all that is needed to reproduce it. Removing either one stops reproduction, though I suspect that may just be due to terminal window size. EDIT: actually you can reproduce it if you just copy the encore.js.map file twice in the same directory. The sourcemap file is extremely long (146k characters) and was presumably all being passed to less? This may be the issue? |
|
Hmm, I still can't produce on either Mac or Linux. Not sure what's going on. I could maybe see if |
|
@danielcompton Do other search tools exhibit this behavior? e.g., |
|
|
$ pwd
/private/tmp/taoensso
$ ls
encore.js.map encore.js.map copy
$ rg cmd | less
# hangs ... |
|
So I'm not quite sure how to move forward on this one. I can't reproduce it so I don't know how to fix it, and I'm not quite sure what debugging steps I would try next. If you're interested in getting into the nitty gritty, then I can think of a few things we can try:
|
|
I ran Err#60 is ETIMEDOUT |
|
@danielcompton Thanks! I'm not quite sure what to make of that, but I'll noodle on it. |
|
Here's a slightly more complete trace from start to finish:
You can ignore the |
|
I can also reproduce this issue on Windows using One thing I noticed is that for small outputs it looks like EDIT: Could it also be very long lines in the output? Both @danielcompton's gist and my private files have a lot of output on a single line. |
|
Ah yes, forgot to mention I can reproduce this with |
|
It turns out that "noodling" on this problem was exactly what was needed. I ended up figuring out how to produce this. Basically, it happens whenever you:
The issue here is that when one thread blocks on writing to This also caused me to realized another shortcoming of the existing iterator. It was running searches for all direct children of a directory in the same thread. So if you ran ripgrep on a single directory with lots of files, you wouldn't get any parallelism! Owch. |
|
Thanks so much for reporting this and digging into it! It was a tricky one! |
|
Thanks for fixing it, you did the hard work :) |
|
Is there a build I can verify this against? |
|
@danielcompton No, you'd have to compile from source. I plan to make a release soon though! |
This is a weird behaviour, and may not be ripgrep's fault, but I thought I'd mention it in case.
I piped rg into less with
rg cmd | lessand looked at the first page of results, left them there and continued working. After a while I noticed that my CPU usage was high and rg was using 200% of my 400% CPU's available. If I run the command withoutlessand time it, it takes around 0.5 seconds, but with less, the CPU sits at 200% continually.If I type
Gthen less goes to the end of the window and thergprocess exits.The text was updated successfully, but these errors were encountered: