ripgrep output is invisible in PowerShell

[nzbart]

When I launch cmd.exe and run PowerShell from within that console window, the output is perfectly legible:

However, if I launch PowerShell and run rg within the default window, it appears that ripgrep renders the file name in blue rather than magenta, making it invisible:

When I tried setting my colour scheme using the PowerShell console configuration window (Alt-Space, P from within the console) so that it matched the cmd.exe settings, I noticed that ripgrep was outputting blue text.

Why would ripgrep change the output colour of the file name in some cases, particularly when the background colour is blue, and is it possible to make this algorithm a bit smarter?

[BurntSushi]

These are the default styles used by ripgrep in all circumstances:

ripgrep/src/args.rs

Lines 706 to 711 in f5a2d02

	let mut specs = vec![
	"path:fg:magenta".parse().unwrap(),
	"line:fg:green".parse().unwrap(),
	"match:fg:red".parse().unwrap(),
	"match:style:bold".parse().unwrap(),
	];

        let mut specs = vec![
            "path:fg:magenta".parse().unwrap(),
            "line:fg:green".parse().unwrap(),
            "match:fg:red".parse().unwrap(),
            "match:style:bold".parse().unwrap(),
        ];

In other words, ripgrep isn't changing anything on you. I can't really explain what you're observing though. Maybe the blend of colors makes it invisible? Maybe Powershell has a more limited color palette and can't display magenta? I don't know.

You might consider trying to experiment with the color settings, which are configurable on the command line:

        --colors <SPEC>...
            This flag specifies color settings for use in the output. This flag may be provided
            multiple times. Settings are applied iteratively. Colors are limited to one of eight
            choices: red, blue, green, cyan, magenta, yellow, white and black. Styles are limited
            to nobold, bold, nointense or intense.
            
            The format of the flag is {type}:{attribute}:{value}. {type} should be one of path,
            line or match. {attribute} can be fg, bg or style. {value} is either a color (for fg
            and bg) or a text style. A special format, {type}:none, will clear all color settings
            for {type}.
            
            For example, the following command will change the match color to magenta and the
            background color for line numbers to yellow:
            
            rg --colors 'match:fg:magenta' --colors 'line:bg:yellow' foo.

and is it possible to make this algorithm a bit smarter?

I would rather have a single set of defaults with no smarts. If those don't work for whatever reason, then I'd like to encourage users to set their own colors with the --colors flag.

[nzbart]

I'm not sure why the colour changes between the consoles, but this is the solution I'm using to work around it thanks to your suggestion.

For anyone else who runs into this issue, this should fix it:

Add-Content $PROFILE -Encoding UTF8 "`r`nfunction rg { rg.exe --colors 'path:bg:white' `$args }`r`n"

[BurntSushi]

@nzbart Oh cool! Could you explain your Add-Content command in more details for folks that aren't familiar with Powershell? I'd love to add something like to the docs.

[nzbart]

That command line adds the following to the current user's PowerShell profile, whose path is stored in the variable $PROFILE:

function rg { rg.exe --colors 'path:bg:white' $args }

Unfortunately, it may not be 100% reliable in all cases depending on how people have configured their profile, so I'd be a bit reluctant to promote it to the official docs.

Something that might be worth adding to the docs is that the simplest way to install ripgrep on Windows may be the Chocolatey package that @dstcruz has kindly created:

• ripgrep package on Chocolatey.org

Chocolatey is an increasingly popular way of installing software from the command-line. For example to install ripgrep:

choco install ripgrep

[BurntSushi]

@nzbart Awesome, thanks for the info! I'll try to improve the docs soon with your helpful feedback. :-)

[DoumanAsh]

@nzbart Note that when you use such alias (through function) rg will not work through pipe
i.e. rg -h | rg hidden will not search output of rg -h but instead will regularly search files in directory.

P.s. the problem is that when you try to pipe output to such function. It will not pipe output to rg.exe

[BurntSushi]

@DoumanAsh is there anyway to work around that and write a functional alias on Windows?

[DoumanAsh]

It took a bit of re-search but so far i found simple solution:

function grep {
    $input | rg.exe --hidden $args
}

$input goes obviously for piping, and it works just fine even if you would have no pipe.

Reference to powershell builtin variables https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_automatic_variables

[nzbart]

@DoumanAsh nice improvement, thanks!

[nzbart]

@DoumanAsh I've found that the function above doesn't seem to work unless piping into it (Windows 10):

> gcm rg | select -expand definition

    $input | rg.exe --colors 'path:bg:white' $args

> set-content -Path test.txt -Value "Something" -Encoding UTF8
> rg thing
> rg.exe thing
test.txt
1:Something
> ls | cat | rg thing
1:Something

Does the test scenario above work for you (run steps in a clean directory)?

[DoumanAsh]

It works for me:

08:15:21 > set-content -Path test.txt -Value "Something" -Encoding UTF8
D:\repos\test
08:15:21 > rg thing
test.txt
1:Something

Maybe it depends on version of powershell?
Which do you use?
Try something above version 2.

You can do it by starting powershell with option -version <num>

I tested on version from 3 to 5 and it works. Cannot go below because my profile wouldn't load :D

[nzbart]

@DoumanAsh That is strange. Could you please confirm that rg in your test above is calling the function and not rg.exe directly? One way to do this is to run gcm rg | select -expand definition.

This is my PowerShell version:

> $PSVersionTable.PSVersion.ToString()
5.1.14393.693

[DoumanAsh]

Opps... you're right, i forgot that i aliased it to grep
Yes, it doesn't work like that. I wonder why i didn't notice it before :(

[DoumanAsh]

Ok, i suspect $input has something even when stdin is empty and therefore rg is trying to read from there regardless.

[DoumanAsh]

Ok, i found work-arond:

function grep {
    $count = @($input).Count
    $input.Reset()

    if ($count) {
        $input | rg.exe --hidden $args
    }
    else {
        rg.exe --hidden $args
    }
}

Basically i convert $input to Array and count number of elements.
And then i reset it back and depending wether there is something inside $input i pipe it to rg

[nzbart]

Unfortunately, it's just much simpler to go back to the Silver Searcher than it is to have to work around this issue on every Windows machine I work on. Below is a comparison of these two tools when run in in the default PowerShell window on a clean Windows 10 install:

As stated in my original post, it does work when starting the default cmd.exe console rather than PowerShell:

Hopefully, the out-of-the-box experience will be improved in a future version for Windows PowerShell users.

As an additional curiosity, I'm unsure why the UTF8 BOM is rendered differently by ripgrep between the two consoles.

[BurntSushi]

@nzbart I don't understand how ripgrep can work on all possible terminal color schemes out of the box. I think what you're saying is, "ag happens to pick a default color that works in my color scheme"?

[BurntSushi]

@nzbart Would #196 solve it? I guess I don't understand why your work-around is so annoying.

[elirnm]

I think the underlying issue here is not that the ripgrep color scheme doesn't match the Powershell scheme, but that the colorization isn't always working properly on Powershell.

"path:bg:yellow" renders with a white background, but not the same white as "path:bg:white". See pictures below.

"path:bg:magenta:" renders with the exact same background color as the default background. This is a blue color, but not the same blue that's produced by "path:bg:blue".

"path:bg:green", "path:bg:red", "path:bg:blue", "path:bg:cyan", "path:bg:white", and "path:bg:black"all render correctly.

The same thing happens with "path:fg:whatever".

Note that this seems to only happen with the powershell.exe executable itself. If I open cmd.exe and invoke powershell within that window, everything works fine.

These were all tested with Powershell 5.0.10586.117 (note: not the most recent version) on Windows 7 with the default Powershell color scheme.

"path:bg:yellow":

"path:bg:white":

"path:bg:magenta" (with the background color of Powershell changed to make ripgrep's colorization visible, and "path:fg:white" set to make the foreground text visible):

"path:bg:magenta" (in the default Powershell color scheme, with "path:fg:white" set to make the text readable. The background color is completely invisible.

"path:bg:blue" (in the default Powershell color scheme, with "path:fg:white" set to make the text readable). Note that this is not the same blue as "path:bg:magenta" gets, and it's (somewhat) readable even on the default color scheme.

[BurntSushi]

@elirnm If you toggle between --colors 'path:style:intense' and --colors 'path:style:nointense', does that help at all?

[elirnm]

--colors "path:bg:yellow" --colors "path:style:intense" --colors "path:fg:black"

--colors "path:bg:magenta" --colors "path:style:intense" --colors "path:fg:white"

[BurntSushi]

Interesting. So I can't really tell if there's an actual bug in the communication with the Windows console. It seems like not? Maybe ripgrep should pick a default color scheme on Windows that fits the default color scheme of PowerShell and call it a day?

[elirnm]

It seems like it might be a bug in powershell.exe, actually, or powershell.exe has a weird idea of colors, because this is happening natively as well.

In powershell.exe:

In a Powershell session invoked from cmd.exe:

[elirnm]

I don't know much about console coloring, but PowerShell/PowerShell#2381 might be relevant. Also https://stackoverflow.com/questions/20541456/list-of-all-colors-available-for-powershell

[DoumanAsh]

I'm using black color sceheme in powershell and it looks fine to me.
It is a bit dark but overall ripgrep's default color scheme is fine.

[elirnm]

@DoumanAsh By "looks fine" do you mean it's readable or do you mean the colors are actually correct? If you execute Write-Host "blah" -ForegroundColor DarkYellow is the output actually dark yellow or is it white? Does Write-Host "blah" -ForegroundColor DarkMagenta produce dark magenta or a dark blue? Are the file paths that ripgrep prints out in magenta or dark blue?

It should obviously be perfectly readable on any shell where the background color doesn't conflict with ripgrep's colors.

[DoumanAsh]

Here is example of my look: http://i.imgur.com/gXU5l00.png
To me it seems to match -ForegroundColor

-ForegroundColor DarkYellow is actual dark yellow and the same for Magenta and Red

[elirnm]

What version of Powershell do you have? $PSVersionTable

[DoumanAsh]

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.14393.0
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.0
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

I'm using latest PS

[elirnm]

I just updated to Powershell version 5.1.14409.1005 and it still doesn't work correctly for me. Both magenta and yellow print the wrong colors. Maybe something with different Windows versions? I don't know.

[nzbart]

@elirnm I think you might have hit upon the problem in your comment above (#342 (comment)) - well diagnosed!

@BurntSushi you asked above (#342 (comment)) whether this was a big problem. I prefer ripgrep over Silver Searcher, but they are both great tools, and one of them works out of the box on multiple machines without having to come up with unsatisfactory workarounds. Each of the PowerShell workarounds we tried above have some kind of deficiency that makes it annoying to use, along with the fact that I work on multiple machines that need the workaround applied instead of just running choco install ag or choco install ripgrep to get up and going.

I still use ripgrep in Vim via Grepper because console colour isn't a problem in this case, and because ripgrep has some very useful features such as understanding .gitignore files and having the --smart-case option.

[latkin]

This is because the default shortcuts for Windows Powershell use a customization of the standard console color palette, redefining "DarkMagenta" to be RGB(1,36,86) and "DarkYellow" to be RGB(238,237,240), then setting the background color as "DarkMagenta." This was done in order to give powershell a custom, distinctive blue color, see http://www.leeholmes.com/blog/2008/06/01/powershells-noble-blue/.

They overwrote dark magenta because it was judged to be rarely used (speculation). Dark yellow was updated because it didn't have enough contrast against the new blue background.

You can see the 2 altered colors when you compare options from Powershell and Cmd:

The overriding of the RGB values is done via the registry:

Upshot is that any tools that write with DarkMagenta foreground are gonna be invisible in standard powershell consoles. IMO this was a total hack from Microsoft, but at this point it's spilled milk.

I think one possible workaround is to check if rg is 1. running on windows and 2. current console background color is dark magenta. That's a very strong indication that you are within a powershell console, and regardless file names will be invisible (either magenta/magenta or "noble blue"/"noble blue"). In this case output file names with intense magenta (which will still look good).

[BurntSushi]

I am open to any easily maintainable hack that fixes this. Even something like, "change the default colors on Windows regardless of shell" would be OK with me I think.

[latkin]

one possible workaround is to check if rg is 1. running on windows and 2. current console background color is dark magenta. ... In this case output file names with intense magenta (which will still look good).

I have this implemented (though I'm new to rust, so hacks, surely). Should I send a PR?

[BurntSushi]

@latkin Sure! I can help you get it into a non-hacky state. :-)