2D view of multiple DepthImages doesn't update with time scroll

[eskjorg]

Describe the bug
2D view with multiple DepthImages doesn't show the correct image when dragging the slider.

But hovering still works, showing the correct pixel for each layer. This indicates that the correct data is there.

To Reproduce

for i, depth_image in enumerate(depth_arr):
    rr.log(
        f"{phase}/depth_imgs/{i}",
        rr.DepthImage(depth_image, meter=1000.0, colormap=Colormap.Viridis),
    )

Then press play or drag slider through time.

Expected behavior
The 2D view should show each corresponding image when scrolling.

Screenshots

This is bad:

screen_rec.mov

This is good:

Desktop
macOS Sonoma 14.5 (23F79)

Rerun version
rerun-cli 0.18.2 [rustc 1.76.0 (07dca489a 2024-02-04), LLVM 17.0.6] aarch64-apple-darwin release-0.18.2 59ff15b, built 2024-08-29T13:55:42Z

[Wumpf]

Interesting find! So actually DepthImage is not supposed to support (transparent) layering at all: this is ever so slightly hinted by the fact that only Image and SegmentationImage come with an optional Opacity component.

This means that a depth image in front will always hide everything behind it. So unless you log new data to the front-most entity, you won't see anything.
Which in turn means that the actual bug here is unfortunately the thing you called out working: Multiple previews should not show the hover interaction is meant to be disabled for fully hidden things (just like a mesh being drawn in front of a point doesn't allow hovering the point).
Note also that the default DrawOrder is stable, but undefined, so you can't be sure which image is in front unless you specify it yourself.

But obviously your intention here is to get information out of several images so I'm wondering what you would like this to look like. Can you elaborate a bit on that?

So far we considered it not meaningful to have transparent layering on depth images - I have a hard time understanding how to get meaningful information out of such a visualization. Typically in those cases you'd either keep logging to the same entity path (then at every point in time you have a different image being shown) or, if all of them should show at once, you'd create more Views via a blueprint.

[eskjorg]

But obviously your intention here is to get information out of several images so I'm wondering what you would like this to look like. Can you elaborate a bit on that?

I was hoping that there was some clever way in which a stack of images would be shown. So they still only take up one "tab" in the viewer. For example another slider separate from time that would scroll among the images here. Maybe I should just stack them into a tensor and log that once. Would that show better?

So far we considered it not meaningful to have transparent layering on depth images - I have a hard time understanding how to get meaningful information out of such a visualization. Typically in those cases you'd either keep logging to the same entity path (then at every point in time you have a different image being shown) or, if all of them should show at once, you'd create more Views via a blueprint.

I see then which 2 alternatives I have. Using DrawOrder for example helped. Thanks!

Note also that the default DrawOrder is stable, but undefined, so you can't be sure which image is in front unless you specify it yourself.

Yes, I've noticed that the draw order here seemed strange but still the same every time when I rerun the code. Why is that?

[Wumpf]

Maybe I should just stack them into a tensor and log that once. Would that show better?

you could do that to get the slider you're looking for, but then you loose all the features that are specific to depth images (like depth cloud generation when viewed in 3D), also the Tensor view has a few shortcomings like more limited hover interaction etc.
Really though the timeline is right now the go-to tool for this kind of interaction. Another orthogonal slider that operates on which entity is shown sounds like a cool idea but a lot more design effort would need to go into making that well defined 🤔
From the pov of the existing systems that would bottom out in "make all entities visible except one, which one controlled by a slider that is somehow tied to entity paths" which is a bit tricky (there's no special concept right now for numbered entitites either, those are just names).

Yes, I've noticed that the draw order here seemed strange but still the same every time when I rerun the code. Why is that?

In a nutshell as long as you don't log draw order, the order is "random but stable". Right now I believe default draw order is determined by alphabetical order of entity names, but this may be subject to change. We can't rely on log order since data may come in different order, and at that point we also don't want to special case on the default timelines to extract this information since there's some paths of data ingestion that don't have them.

---

Not entirely sure on the actionable things here now: I created a separate bug for the unexpected multihover. But beyond that we'd need a whole new ui concept here to support this. Not saying that wouldn't be a good thing to have, it's just that I can't quite yet get my head around it.

[eskjorg]

Thanks for all explanations!
I say don't invest time in any new UI for my sake, I barely look at those depth images, just thought I found a bug.
( Would be much more interested in this #2499 ! )

Fine to close this issue now.

[Wumpf]

Alright, let's close this for now then. Definitely a design direction we'll have to get back to though 🤔
cc: @gavrelina