Support --color-moved

[dwijnand]

Git recently added --color-moved support, to colour moved code: https://stackoverflow.com/a/48166435/463761

Would it be possible to add support for that in delta?

[dandavison]

Thanks @dwijnand, I didn't know about this feature in git. Here are some thoughts, let me know if this sounds right to you.

From a quick look at the docs and play around with the feature, it looks to me like git uses color alone to distinguish "moved" code from "changed" code: i.e. git does not, for example, use special - and + markers for moved blocks.

Now, delta cannot assume that it has access to the original color codes from git. EDIT 2021: That statement is incorrect. The colors are always available when delta is git's pager, and they are available if git is piped to delta if the user supplies --color=always So that suggests that we would have to re-implement git's algorithms for moved code detection. That in itself sounds like a blocker: it would be fragile to attempt to track the details of their algorithm.

However, if git emitted (or could be configured to emit) special +/- markers for moved code blocks then the situation would be entirely different and we could definitely have delta apply special colors to them. I do wonder whether it would make sense for git to do that. After all, without that, nothing can programmatically obtain the information from git output.

Currently delta only uses the input text and does not read anything from the git repo (i.e. delta does not use a library like libgit2, whereas e.g. bat does). So in fact delta doesn't know or care what repo the input comes from and may not even have access to that repo. I don't know if something like libgit2 might supply information about moved code blocks now or in the future.

In addition to being fragile to reimplement git's algorithms it would also be a potential perfomance hit: delta is currently implemented in a streaming fashion: as it moves through the input, it only stores a small amount of local context in memory. But for delta to do moved code detection, it seems to me that in the worst case it would have to read the diff for one file into memory and do some computation before starting to emit any output (e.g. if it encountered a - block at the beginning of the diff, it would need to read to the end of the file before knowing whether to color that as deleted or moved. I don't know how often that would result in a noticeable performance hit for delta users, but it would be a non-trivial code change. Git has the repo available to it and thus probably can do this more efficiently.

What do you think, am I understanding the situation correctly?

[dwijnand]

Thanks for the detailed write-up! I didn't spend time thinking about it too much, but I can share my thoughts based on your analysis.

Now, delta cannot assume that it has access to the original color codes from git.

What if:

1. it uses the colour codes you've separately configured in delta; or
2. it uses the .git/config/~/.gitconfig-configured colour codes, provided it can access them; or
3. it just uses git's defaults (or bombs)

?

[dandavison]

Ah-ha, I see. So currently, delta ignores any ANSI color codes in the input it receives it from git. But we could in fact inspect the color codes coming from git, and compare them to what we believe git is using for moved code, and thus figure out when we are in a special moved code block rather than a normal added/deleted block.

This wouldn't work if someone were to do git diff | delta because by default git does not write color codes into a pipe, but it would work for the most important case where delta is configured officially as the git pager via git config's core.pager variable.

Does that sound right?

[dwijnand]

Yeah, that was the setup that I had in mind. And you're right to point out the piping caveat, for the alternative usage.

[dandavison]

OK great, this seems very doable. We can always add an option to disable it / make it non-default if the color-sniffing is unreliable for some reason.

So the next question might be: what is a good way for delta to display moved code blocks? Is it simply a case of adding a third and fourth background colour? Any advice on the actual design / colour selection would be appreciated; I found it quite hard to come up with good defaults, especially for dark terminal backgrounds, and that was with just two background colours. cc @nkouevda @fdcds.

[dwijnand]

Yeah, that might be a challenge, given you're already adding both syntax highlighting and within-line highlighting... I don't have any suggestions, sorry.

[0xC0FFEE]

So the next question might be: what is a good way for delta to display moved code blocks? Is it simply a case of adding a third and fourth background colour? Any advice on the actual design / colour selection would be appreciated; I found it quite hard to come up with good defaults, especially for dark terminal backgrounds, and that was with just two background colours. cc @nkouevda @fdcds.

Choosing the right™ colors is always hard (spoiler: there will always be someone saying that color X is better). I've personally been using some variation of red/green for moved code for quite some time and I'm total used to this by now. So much, that currently this prevents me for using delta on a day to day basis :-).

Having said that, I would make the colors configurable and do not worry too much about the defaults in the beginning. IMHO defaults can be optimized after this feature has seen some real world tests and once they are good enough, the feature can be enabled by default.

Some possibilities to indicate moved lines could be:

• Make the background color configurable
• Prefix lines with some marker (e.g. +N/-N where N is the number of the moved block in this hunk)
• Maybe disable syntax highlighting for moved code and make the foreground color configurable
• Using the same background color as added/deleted lines, but draw the whole line in italics/underlined/...

[zx8]

This is the single missing feature holding me back from making the final switch to git-delta – in every other regard, git-delta seems far superior to any of the alternatives from my initial testing, but I've come to rely on quickly identifying and filtering out moved lines so much that I keep having to revert back to diff-so-fancy, solely for --color-moved support.

[dandavison]

Hey @zx8, thanks for that. I did see your comment when you made it a couple of weeks ago and I have started work on this. Hopefully there'll be something soon!

[dandavison]

Support for color-moved is now in the master branch and I'll release it soon. Meanwhile, if anyone's able to build on master and report back that would be fantastic.

Below are some implementation notes: the TL;DR is that Delta supports color-moved by emitting moved lines with exactly the same color styling as they have under Git. This means that all Git's nuances associated with the values of --color-moved and --color-moved-ws are available in Delta, but that we are unable to apply syntax highlighting to moved lines.

Since this issue was opened a long time ago, I'll just mention that Delta has evolved quite a bit since then. In particular, the most convenient way to configure Delta is now with a [delta] section in ~.gitconfig, and Delta supports git style strings everywhere. There are also various new features such as line numbers, side-by-side view, and diff-highlight/diff-so-fancy emulation. (README.md).

--

I considered 3 ways to implement this (recall that the only way for Delta to infer that Git has identified a line as moved is by inspecting ANSI color escape sequences):

1. If the line does not have the ANSI escapes expected for a typical removed/added hunk line, then emit the raw line with its ANSI escapes intact.
   This is what I chose, but it means we cannot syntax-highlight moved lines.

2. If the line does not have the ANSI escapes expected for a typical removed/added hunk line, then interpret it as a moved line and style it with a special Delta style for moved lines.
   This would give us syntax-highlighting for moved lines, but it loses Git's color-moved option nuances and it is fragile.

3. If the line has ANSI escapes expected for one of Git's 8 moved color styles, then identify it as such and apply a corresponding special Delta style.
   This would give us syntax-highlighting and all Git's color-moved nuances but it is complex and fragile.

(2) is fragile in that Git might emit colors other than the canonical diff.old/diff.new for hunk lines in situations for reasons other than color-moved. I think (?) it doesn't today, but it seems plausible that a future git feature will do this.

(3) is fragile in that a user might configure another git style to be the same as one of the moved styles. If that were so, Delta would have no way to avoid the ensuing error.

A challenge faced by all 3 options is that two things influence the colors emitted by Git: color values configured in gitconfig, and git's hard-coded default colors. Since a few months ago, Delta does read values from gitconfig, so that's not a blocker. Git's hard-coded color defaults are (AFAIK) private and could change with any release.

So I went with (1): if Delta sees something that doesn't look like a typical removed/added hunk line, it emits the raw line unaltered. For this it was still necessary to bake into Delta the values of Git's default red/green colors for removed/added lines. But if you have set color.diff.{old,new} to something different in gitconfig, Delta will spot that.

If for some reason this color-sniffing goes wrong, it's somewhat disastrous: Delta won't give any of its usual output. So, color-sniffing is the default :) There is however a kill-switch: inspect-raw-lines = false.

As one additional safety measure, if Delta sees the standard red/green foreground styles, it will always assume this is a typical removed/added hunk line. That's true even if you've actually set one of the colorMoved styles to red and set diff.old to something else. I did that out of paranoia; in principle it's unnecessary, so let me know if it's restrictive.

[dandavison]

By the way, the implementation notes above were in mainly in case anyone has ideas about how Delta can support color-moved better, so please feel free to suggest improvements / ways we could support syntax highlighting, etc.

cc @navarroaxel @Kr1ss-XD @Ryuta69 if you're able to run Delta from the master branch in case you encounter bugs in the color-moved support that would be great (I just found a bug, triggered by using the blink attribute in color.diff.{old,new}).

In case anyone else feels like doing some testing, configuration would be something like

[diff]
	colorMoved = default
	# colorMovedWS = allow-indentation-change  # <-- can experiment with this, see #144

[color "diff"]
    # Delta should still work correctly, for whatever nonsense you put here!
    old = bold "#aabbcc" brightblack blink
    new = strike ul 22 "#ddeeff" dim

[dandavison]

In order to make the color-moved styles harmonious with your Delta styles, you may want to make the background colors match. To find out the background colors that Delta is using given your current setup, you can use delta --show-config, e.g.

delta --show-config | grep style

Based on that, I'm using something like the following

[color "diff"]
    oldMoved = bold brightmagenta "#ffe0e0"
    oldMovedAlternative = bold brightmagenta "#ffe0e0"
    newMoved = bold brightblue "#d0ffd0"
    newMovedAlternative = bold brightblue "#d0ffd0"

(Although I appreciate that most people will be using a dark background and so the appropriate background colors will be different.)

[navarroaxel]

Hey @dandavison, why in the first part of your screenshot, line added 35, the pub and bool keywords are highlighted?

[dandavison]

Hey @dandavison, why in the first part of your screenshot, line added 35, the pub and bool keywords are highlighted?

That's a good question, I was wondering the same. I've double-checked, and it is not Delta's fault; it's to do with Git's color-moved behavior. If we set diff.colorMoved to zebra, or default, or blocks, then Git outputs that line as a normal removed/added pair, not as a moved pair. But if we set diff.colorMoved=plain, then Git colors it as moved. I haven't fully understood the different Git colorMoved modes yet, does anyone know whether that is all proper and expected behavior for Git?

But in any case, Delta has no ability to second-guess Git here: the rules are that if the colors received from Git are not what is expected for a normal removed/added line then Delta outputs the line raw. But l.35 is colored as expected for a normal added line, so Delta does all the normal Delta processing and rendering for it.

E.g.

HOME=/dev/null git -c 'core.pager=less -R' -c 'diff.colorMoved=default' show c1a49f46f src/config.rs

[navarroaxel]

Oh, I didn't notice! By default, git detects blocks of moved text of at least 20 alphanumeric characters (mode: default, zebra, blocks) but the plain mode picks up any moved line, but «it is not very useful in a review to determine if a block of code was moved without permutation». (quoting the git doc).

So, these highlighted keywords are the expected behavior.

[dandavison]

Thanks @navarroaxel, I see, I didn't read that nearly carefully enough! So if we exclude non-alphanumerics then I guess that line is less than 20 characters.

[dandavison]

This is released now (delta 0.4.0). Thanks everyone for suggesting and discussing this.

[chtenb]

3. If the line has ANSI escapes expected for one of Git's 8 moved color styles, then identify it as such and apply a corresponding special Delta style.
   This would give us syntax-highlighting and all Git's color-moved nuances but it is complex and fragile.

Any chance we would see this option implemented in the future? If yes, I will open a feature request for this. The functionality would be absolutely killer. I would personally like to configure the delta theme to apply the color-moved color to the background, and syntax highlighting to the foreground, similarly to added/deleted lines in many themes (like the Github one for instance).

Although, one must take care that the syntax highlighting doesn't conflict with any of the color-moved colors in that all combinations must remain readable. Perhaps delta should just support configuration of a single style pair for color-moved, just like the plus/minus style pair. All recognized moved lines would then be formatted according to that style, regardless of which color style git assigned to a particular line.

[unphased]

I think that the zebra concept is well-reasoned since it is important to indicate when a single section is moved into two separate sections, where that original section was broken up. But, I do think that instead of having to make it alternate with a zebra approach all we would need is a way of indicating a boundary.

I agree that having syntax on the moved chunks would fully complete the staggering masterpiece that is Delta.

[unphased]

Setting aside the desire for syntax-highlighted moved regions:

It's visually confusing to have moved content receive a foreground color: it should be background colors like the plus/minus colors, for semantic consistency.

In adjusting the colors, because of the approach taken (and I'm very grateful for the detailed explanation above) since the colors are used by delta verbatim, the setting would then make a non-Delta diff look similarly inconsistent, by introducing background color to a diff only in moved sections. I actually think that is fine by me given how rarely i use the default vim output... but it does happen frequently to want to use a faster diff when traversing long distances in a git log.

Anyway, this just highlights the drawbacks of this approach no.1.

---

I want to ask a question: how do we feel about independently computing the moved regions, I think there are situations when it could be useful to see moves smaller than 20 characters (though gotta admit the choice of 20 is fairly solid). In particular I think that being able to indicate the specific movements from and to would be useful. What I mean is for the output to tell us for each moved chunk which place it was added back in. For a terminal interface the only ways to do this association might be to index them. Maybe could background-color code them in side-by-side mode, I feel like it does not have to violate the convention noted above for background color indicating a change, since minus is red and plus is green, I think shades of blue are suitable to indicate movement. The problem with this is that it falls flat in unified (non side by side) mode. Potentially, brightness/saturation can be used to map the chunks, and we still keep two overall colors, they can be magenta (movedFrom) and cyan (movedTo).

Dunno how out there this idea is, but maybe it could be option number 4.

[unphased]

I take it the programming language could only be inferred from the filename, so it may be deficient if I have say python scripts or shell scripts that do not have a file extension?

[dandavison]

I take it the programming language could only be inferred from the filename, so it may be deficient if I have say python scripts or shell scripts that do not have a file extension?

Yes, that's right. I speculate here #761 (comment) about creating a Rust port of Github's linguist tool that infers language from file content. (There are some special cases though -- e.g."Makefile" is recognized as such. These rules come from the collection of sublime language syntax definitions maintained by the bat project which delta uses).

[dandavison]

sometimes we want to scan down fairly low in the git log, so then it does become a factor how quickly it can be processed

Right, I use git log -p a lot myself, but hadn't actually made the connection with performance. That's a good point to bear in mind -- perhaps it is a reason to improve delta performance. git log -p followed by / is a good lazy alternative to remembering how to use git grep on history!

[unphased]

Actually on the linux repo I did not see any evidence that it is slower. Probably that is bottlenecked on the sheer size of that repository. On this much smaller but still pretty big (230MB) repo I'm using at work:

With cpipe comparing time git log -p | cpipe -vr -vw -vt > /dev/null to time git log -p | delta | cpipe -vr -vw -vt > /dev/null

  in:  18.499ms at    6.8MB/s (   2.2MB/s avg)   21.0MB                  │ out:   0.001ms at  122.1GB/s ( 135.3GB/s avg)   92.1MB
 out:   0.001ms at  122.1GB/s ( 149.7GB/s avg)   21.0MB                  │thru:  37.542ms at    3.3MB/s (   3.7MB/s avg)   92.1MB
thru:  18.516ms at    6.8MB/s (   2.2MB/s avg)   21.0MB                  │  in:  32.446ms at    3.9MB/s (   3.7MB/s avg)   92.2MB
  in:   0.151ms at  827.8MB/s (   2.2MB/s avg)   21.1MB                  │ out:   0.001ms at  122.1GB/s ( 135.3GB/s avg)   92.2MB
 out:   0.000ms at    infGB/s ( 150.6GB/s avg)   21.1MB                  │thru:  32.468ms at    3.8MB/s (   3.7MB/s avg)   92.2MB
thru:   0.159ms at  786.2MB/s (   2.2MB/s avg)   21.1MB                  │  in:  22.573ms at    2.7MB/s (   3.7MB/s avg)   92.3MB   (bsize=62929)
  in:  16.747ms at    7.0MB/s (   2.2MB/s avg)   21.2MB   (bsize=122468) │ out:   0.002ms at   29.3GB/s ( 134.9GB/s avg)   92.3MB
 out:   0.002ms at   57.0GB/s ( 149.2GB/s avg)   21.2MB                  │thru:  22.596ms at    2.7MB/s (   3.7MB/s avg)   92.3MB
thru:  16.766ms at    7.0MB/s (   2.2MB/s avg)   21.2MB                  │git log -p  9.84s user 0.13s system 40% cpu 24.633 total
git log -p  9.60s user 0.11s system 99% cpu 9.721 total                  │delta  16.28s user 0.48s system 67% cpu 24.697 total
cpipe -vr -vw -vt > /dev/null  0.00s user 0.02s system 0% cpu 9.721 total│cpipe -vr -vw -vt > /dev/null  0.03s user 0.39s system 1% cpu 24.697 total

It's only marginally slower. Probably no hugely significant gains to be had. It's 2.5x slower but produced 4x more output so it has a higher actual bitrate fwiw (that's also the in and thru numbers, the out is just seemingly a measurement of CPU cache bandwidth)

Conclusion: As usual for Rust, Delta is already blazing fast, async would be the cherry on top, def would keep up no problem with the input stream even on a quad core.

[unphased]

Yes, that's right. I speculate here #761 (comment) about creating a Rust port of Github's linguist tool that infers language from file content. (There are some special cases though -- e.g."Makefile" is recognized as such. These rules come from the collection of sublime language syntax definitions maintained by the bat project which delta uses).

@dandavison Could we consider parsing the first line of a file for a shebang as a last resort if it is an unknown filetype to check what language it may be? This would address the main situation in which it fails, and is an ideal 80-20 (but more like a 98-2 here) win.

[dandavison]

Could we consider parsing the first line of a file for a shebang as a last resort if it is an unknown filetype to check what language it may be?

Yes, I think that makes sense. (Of course we can't rely on line 1 being present in the diff, so it changes the rule above about delta only operating on stdin. And implementation note: if the path doesn't exist naively because the user isn't in the repo root, we can try to form the correct path using

delta/src/config.rs

Line 273 in 57325f9

cwd_relative_to_repo_root: std::env::var("GIT_PREFIX").ok(),

and

delta/src/handlers/diff_stat.rs

Line 44 in 57325f9

pub fn relativize_path_in_diff_stat_line(

)

PRs welcome!

[chtenb]

Cool, thanks!

[chtenb]

When this gets released I will make some themes that take this into account. Would it be appreciated if I made a pull request to include them in the defaults?

[dandavison]

Would it be appreciated if I made a pull request to include them in the defaults?

Yes, that would be much appreciated. Here's a quick summary of relevant recent changes that I'll release soon:

Here's an example of using map-styles to assign delta styles (one with syntax highlighting) to the raw color-moved styles output by git. This should allow all git's color-moved options to be rendered using delta styles:

[delta]
    features = my-color-moved-theme

[delta "my-color-moved-theme"]
    git-moved-from-style = bold purple     # An ad-hoc named style (must end in "-style")
    
    map-styles = "my-color-moved-theme.git-moved-from-style => red #cccccc, \
                  bold cyan => syntax #cccccc"
                  
    # we could also have defined git-moved-to-style = bold cyan

To make use of that, you need to know that git is emitting "bold cyan" and "bold purple". But that's not always obvious. To help with that, delta now has a --parse-ansi mode. E.g. git show --color=always | delta --parse-ansi outputs something like this:

As you see above, we can now define named styles in gitconfig and refer to them in places where a style string is expected. We can also define custom named colors in git config, and styles can reference other styles; see the hoopoe theme for an example:

[delta "hoopoe"]
    green = "#d0ffd0"  # ad-hoc named color
    plus-style = syntax hoopoe.green  # refer to named color
    plus-non-emph-style = plus-style  # styles can reference other styles

Additionally, we can now use the 140 color names that are standard in CSS. Use delta --show-colors to get a demo of the available colors, as background colors to see how they look with syntax highlighting:

As always, if anyone's able to test out any of the changes by building the master branch that would be great.

[chtenb]

It took a while but I've finally been able to dive into it and dabble with the possibilities. I've currently arrived at the following generic solution.

    dark = true

    minus-style = syntax "#311111"
    minus-emph-style = syntax "#441818"
    plus-style = syntax "#112c1a"
    plus-emph-style = syntax "#1d3f27"

    map-styles = \
       bold purple => syntax "#311128", \
       bold cyan => syntax "#11312d", \
       bold blue => syntax "#23152e", \
       bold yellow => syntax "#222f14"

    zero-style = syntax
    whitespace-error-style = "#aaaaaa"

This is a screenshot from my VSCode. The upper part shows a little bit of my editor view, while the lower part is a git diff output from the integrated terminal.

What do you think @dandavison, would something like this be useful to include as a default feature? I've chosen the colors to be fairly generic and usable across dark themes. I can make one for light themes as well.

[dandavison]

Hi @chtenb, thanks! I tried it out and it's really nice. I'd love to see a light theme counterpart also. Please do add your themes to the collection. It's fairly straightforward -- you just add it to the file themes.gitconfig in the delta repo and make a PR. There are a few instructions to follow in the file, e.g. regarding the mandatory biological theme names :)

What should diff.colorMoved be set to for this theme?

[dandavison]

Including some themes in the distributed executable could make sense also -- but hasn't been implemented yet.

[chtenb]

What should diff.colorMoved be set to for this theme?

That doesn't matter I think. I've set it to default, but anything works.

Also, I would omit settings like linenumbers and commit-decoration-style from this theme, as they are completely orthogonal to the background colors. This means that one can mix and match between sets of features. I notice that the currently included themes are monolithic in this regard.

regarding the mandatory biological theme names :)

Perhaps something related to a zebra would do the job 🦓

[chtenb]

Here is the light variant.

[delta "zebra-light"]
    minus-style = syntax "#fbdada"
    minus-emph-style = syntax "#f6b6b6"
    plus-style = syntax "#d6ffd6"
    plus-emph-style = syntax "#adffad"

    map-styles = \
       bold purple => syntax "#feecf7", \
       bold blue => syntax "#e5dff6", \
       bold cyan => syntax "#d8fdf6", \
       bold yellow => syntax "#f4ffe0"

    zero-style = syntax
    whitespace-error-style = "#aaaaaa"

PR incoming.