transparent option for rasterize?

[rsignell-usgs]

When I use datashade I get the transparent background that lets me see the WMTS tiles

but when I use rasterize, I get grey instead of transparent

and I can't figure out how to fix it.

I did try to figure this out by googling and looking in docs and code, but failed... 😞

[ocefpaf]

Same here, is there an option to set the gray to transparent?

[jbednar]

datashade is just a shorthand for rasterize followed by shade. shade uses Datashader to do the colormapping from within Python (server side), while the default is to use Bokeh to do the colormapping in JavaScript (client side). So you can presumably make it work by doing shade(rasterize(...)), but I suppose you're asking for the Bokeh/JS-based colormapping to work the same as the datashader/Python colormapping, which I agree seems appropriate. I believe that doing so wasn't previously possible due to a problem with Bokeh's handling of special color values, but now that's been fixed in Bokeh (bokeh/bokeh#6756) maybe we need to declare that "transparent" is the color to use by default for NaN? @philippjfr?

[philippjfr]

A transparent background is actually the default when using GeoViews, i.e. if you were to switch to gv.TriMesh on the input to rasterize it would return a gv.Image (as long as you are using master of hv and gv), which uses a transparent color for NaN values. This was a deliberate choice because when plotting non-geographic data you often do want to see the nan values, but I'd be happy to have this discussion again.

To override the default NaN color to be transparent you can set clipping_colors={'NaN': (0, 0, 0, 0)} as a plot option. This will be covered properly in the new "Styling Plots" user guide being added in #2483.

[jbednar]

I would think that the default should be transparent whether or not using GeoViews; is there a reason it shouldn't be? Transparent allows grid lines, etc. to show through, which can indicate missing values much better than any single color can even when there isn't an underlying map to let show through.

[ocefpaf]

This was a deliberate choice because when plotting non-geographic data you often do want to see the nan values

Makes sense.

To override the default NaN color to be transparent you can set clipping_colors={'NaN': (0, 0, 0, 0)} as a plot option. This will be covered properly in the new "Styling Plots" user guide being added in #2483.

I'm fine with that. No need to change the defaults as long as it is properly documented. 👍

[rsignell-usgs]

Cool. This works:

%%opts Image [colorbar=True clipping_colors={'NaN': (0, 0, 0, 0)}] (cmap=palettable.cubehelix.perceptual_rainbow_16.mpl_colormap)
%opts WMTS [width=700 height=400]
tiles = gv.WMTS('https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{Z}/{Y}/{X}.jpg')
points = gv.operation.project_points(gv.Points(verts, vdims=['z']))
tiles * rasterize(hv.TriMesh((tris, points)), aggregator=ds.mean('z'),  precompute=True ) 

Thanks guys!

[jbednar]

Ooh; pretty!

This was a deliberate choice because when plotting non-geographic data you often do want to see the nan values

There's probably some example that would make this clearer to me, but I can't quite appreciate how a grey default color allows us to "see the nan values" in a way that actual transparency does not. And I very much do see how using transparency would be useful by default, to allow images to have a non-rectangular shape. Some good counterexamples could convince me, but I'm definitely still feeling that NaN should be transparent by default.

In any case, can we make it simpler to set this option by defining a special clear or transparent color, to allow it to be clipping_colors={'NaN': 'transparent'}? It's hard to guess that 0,0,0,0 means "transparent".

[rsignell-usgs]

I agree with both these ideas:

• that the default for NaN be transparent
• that clipping_colors={'NaN': 'transparent'} is more friendly, and could be used by those of us who like to be pedantic instead of relying on defaults. 😃

[philippjfr]

@jbednar You are usually the first person to point out that a white background and fire or viridis colormaps (which are usually our defaults) are not good combinations, which is exactly what you'll get if you activate transparency for nans. I personally don't have major objections, but based on our past discussions about colormaps it seems like you would.

Definite +1 from me on aliasing 'transparent' to (0, 0, 0, 0).

[philippjfr]

Seems like there is pretty broad consensus on this so I'm going to implement it in #2483, unless you change your mind @jbednar.

[jbednar]

Sigh; you're right. Using a gray background like mpl used to do would alleviate that issue, but it makes for ugly plots and reduces the dynamic range available when plotting lines and points, so that's a non-starter. Plus some important colormaps (greys!) include grey as a value. So many considerations to try to balance against each other!

Given that the default background is white and that many, many people will use white backgrounds, probably the safest thing is just to make sure that none of our default or generally suggested colormaps include white. People can explicitly choose something like "fire" if they want to have the maximum dynamic range, but it's probably better to prioritize avoiding misleading plots if there might be missing values.

Viridis doesn't actually include white, so there's no issue there. "Hot" and "fire" (colorcet linear_kryw_0_100_c71) do include white, but colorcet actually has one just like fire but without white (or pure black), namely linear_kry_5_98_c75:

Given that HoloViews hasn't actually switched the default colormap away from "hot" yet, I'd argue that "fire" maybe isn't as good a default as this other one, for the reasons you outline. We'd need a name for this slightly muted version of fire, if we use it as a default. I guess Jean-Luc's suggestion "ember" is still available, as we never used that?

In any case, viridis, fire, and ember (if we went with that name), as well as plasma, inferno, magma, and cvidis from matplotlib all are already incompatible with white backgrounds; each of them map increasing values to colors more and more perceptually similar to white:

Specifically, all of them have luminance increasing over the colormaps, using white or yellow for the highest values. So none of them are appropriate as a default colormap if any of the underlying white background is allowed to show through.

I think that the final conclusion is that any such default ought to be the "reversed" versions of one of those colormaps, starting from a value similar to (but distinguishable from) white, and becoming less and less like white as the values increase. So, "ember_r" as a default in holoviews 2.0? One downside is that it's a perceptually quite obvious breaking change from holoviews 1.x, given that "ember_r" is qualitatively the reverse of "hot", but I don't really see a way around that. Another downside is that "fire" and "ember" have a very natural interpretation as "intensity (of heat)" increasing from a black background, and much less so when increasing from a white background, but again I don't see any way around that given that people want a white background by default. In any case, continuing to use a default colormap that is incompatible with a white background, when white backgrounds are the default, seems untenable.

[jbednar]

In any case, @philippjfr, I do still think that NaN ought to be transparent by default. So I'm happy for you to make that change in #2483. As soon as you do, though, it becomes even more urgent to change the default colormap away from "hot", which then becomes even worse as a default than it was before.

[philippjfr]

Okay 'transparent' is now an alias, and Raster/Image/QuadMesh default NaN color is now 'transparent'.

[jbednar]

Great, thanks!

[jlstevens]

All sounds good to me though is there a shorter word we could use than 'transparent' that is clear and means the same thing?

[jbednar]

I think 'clear' is the other obvious option, but it's not quite as obvious...

[jlstevens]

Any harm in supporting both aliases? Then users can decide whether the short name is fine or not.

That said, generally I don't like aliases so I think if we have to pick one, my vote is for 'clear'. Explicitly using(0,0,0,0) is actually shorter than transparent!

[jbednar]

Personally, I don't care about shorter in this case, I care about clearer, and transparent is vastly clearer than (0,0,0,0) and slightly clearer than clear (no pun intended).

[jlstevens]

I don't agree, that is totally obvious as an RGBA tuple with zero alpha. The tuple is totally unambiguous unlike any string name. Also, in general image processing (1,1,1,0) is not the same as (0,0,0,0) and 'transparent' doesn't tell you what you have (or any color in between).

[jlstevens]

In other words, I would be inclined to use 'clear' over the tuple but never 'transparent' over the tuple. But maybe that is just my personal preference...

[philippjfr]

Also, in general image processing (1,1,1,0) is not the same as (0,0,0,0) and 'transparent' doesn't tell you what you have (or any color in between).

Does that matter? Any RGB value with zero alpha should render identically, and 'transparent' makes no claim about the color just that it renders as transparent.

[jbednar]

I think the key here is your phrase "as an RGBA tuple". Yes, (0,0,0,0) is obvious as an RGBA tuple. But that's not the context here; what the user sees is clipping_colors={'NaN': (0, 0, 0, 0)}. Here, I don't think it's at all obvious that the 0's are R's or A's. A suitably informed expert user can infer that, but I'd argue it's far from self evident. Whereas 'transparent' is self evident about the intent. At least in this case there's no need to distinguish between all the various visually identical hues that can be considered transparent, and in fact not bringing that issue up is a strength of 'clear' or 'transparent' compared to the literal RGBA tuple.

[jlstevens]

Does that matter? Any RGB value with zero alpha should render identically...

That pixel yes. But neighboring pixels with non-zero alpha can be tainted by the R,G and B components if resampling. I don't think this problem crops up with anything we do right now but this it generally known that for image processing, the color channel matter even if the alpha channel is zero.

I'm not convinced this case is worth adding an alias for so I am happy that this name is only currently processed in the context of clipping_colors.

Anyway, I can see that I will be overruled on this issue and I don't feel strongly enough about it that I would reject a PR because it includes this alias.

[philippjfr]

The PR has now been merged.

[jbednar]

Bokeh has now added transparent as an alias for color (0,0,0,0): bokeh/bokeh#10576

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.