Excessive logging from new mesh_picking backend

[rs017991]

Note: This is in regard to the new 0.15.0-rc.1 release candidate.
I understand that this is a pre-release, and considered waiting to see if the official release notes might shed light on this behaviour,
but thought you might want to know about the rough edges sooner than that.

The Issue

After updating to 0.15.0-rc.1, I get about a thousand of these every second:

ERROR bevy_picking::mesh_picking::ray_cast::intersections: Invalid intersection check: `TriangleList` is the only supported `PrimitiveTopology`

I am indeed using an alternate PrimitiveTopology, though I am not yet utilising any picking features within my code.

Each of the following is unclear to me:

1. Why it needs to be logging this for every function call (or even every frame), vs using error_once!
2. Why it is considered an error and not a warning (given that it is an optional feature)
3. Why it is attempting to compute so many intersections (given that I'm not requesting such information in my systems)
4. Why it even matters which PrimitiveTopology is used (the other variants work just fine elsewhere in Bevy)

Reproduction

You can reproduce this behaviour with the regular 3d/lines example, with one tweak:

1. Change one or both usages of RenderAssetUsages::RENDER_WORLD to RenderAssetUsages::default()
2. Run the example and move your mouse near the lines

[cart]

Looking into this now

[cart]

I think a core issue here is that we're computing mesh intersections across all spawned meshes by default. The performance tradeoff there is not what most people will want in actual games. Ex: when spawning a complex GLTF scene, pretty much nobody wants raw-mesh intersections.

The "pick any mesh by default" behavior really only makes sense in editor scenarios, which is not what our defaults should be optimizing for.

Physics collision shapes are generally used for in-game picking (similar in concept to SimplifiedMesh in our picking impl). That is already one form of opting in. But when a game developer is building a "click on something" feature, they will define a specific layer / mask to narrow the considered colliders down (prior to the computation of the intersections). This is another layer of opt-in explicitness.

At a bare minimum, we should disable the current mesh picking behavior by default. But I'll assert that the current picking behavior (a global PointerHits event queue) has very limited practical applications. What Bevy app developers will want in general for 3D use cases (and what we should push them toward) is a way to explicitly add collider types to their entities (generally simple / easy to compute shapes like AABBs, spheres, capsules, etc with actual meshes being special exceptions), define the layer those types live on, on-demand request a raycast, and then receive the set of matching items in their own collection. Some use cases might benefit from a global "what did I just click across all mesh types" feed, but I suspect, given the diversity of what might be rendered to the screen (and how selection demands will change based on game context), I'm not sure if its even a path we should be pushing people toward.

A generic PointerHits (or picking interaction) event across all types isn't going to cut it for most applications, both in 2d and 3d. I think its most practical application is debugging and editor scenarios. And even in those cases, I think finer and more intentional granularity would be good. It seems valuable to provide a way to "picking style" trigger observers for click and hover events (ex scenarios like "I want to write an observer for click/hover events when a 3d player collider is hovered"), but I think we need "scoping" to be an integral part of the system.

[cart]

A generic PointerHits (or picking interaction) event across all types isn't going to cut it for most applications, both in 2d and 3d

I'll clarify that it is much more likely to be useful in 2D, and that the cost of enabling it by default would be low enough to be worth considering / people will even expect this to be available by default.

[NthTensor]

In terms of "pick any mesh by default behavior" I'm fine to have that off by default for 3D. The functionality is there if they want it, and should be easy enough to find an enable.

What Bevy app developers will want in general for 3D use cases (and what we should push them toward) is a way to explicitly add collider types to their entities (generally simple / easy to compute shapes like AABBs, spheres, capsules, etc with actual meshes being special exceptions), define the layer those types live on, on-demand request a raycast, and then receive the set of matching items in their own collection.

I agree, I do think this is probably going to be important. We could do a lot more to support this, and I know @Jondolf wants to. But I'm not sure I would use picking for that, that's just raycasting. The mesh-picking backend is intentionally very cut down and narrowly scoped, it's basically just intended for use in the examples (where it's very handy) and the editor prototypes (where it's basically required for a bunch of core interactions).

Just so we're all on the same page, PointerHits aren't really something users are supposed to consume. These events feed the HoverMap which tracks pointer hover state, and then this drives the Pointer<E> events (see the pointer_events system for more information).

So when you say

Some use cases might benefit from a global "what did I just click across all mesh types" feed, but I suspect, given the diversity of what might be rendered to the screen (and how selection demands will change based on game context), I'm not sure if its even a path we should be pushing people toward.

and

I think finer and more intentional granularity would be good. It seems valuable to provide a way to "picking style" trigger observers for click and hover events (ex scenarios like "I want to write an observer for click/hover events when a 3d player collider is hovered"), but I think we need "scoping" to be an integral part of the system.

I'm a little confused, because bevy_picking to me seems much closer to the latter than the former, and the mesh-picking back-end is a core part of making that higher level api run. Following your example, if you want to attach some on-click functionality to all player colliders, you can do something like the following.

world.observe(|trigger: Trigger<Pointer<Click>>, player_coliders: Query<&PlayerColider>| {
    if let Some(colider) = player_coliders.get(trigger.target()) {
        // add my logic to coliders here
    }
});

Please let me know if I'm misunderstanding your point.

[NthTensor]

Thinking about it, I'd be theoretically OK having basically all bevy_picking back-ends off by default. Maybe not UI.

[aevyrie]

@cart I think you might be fundamentally misunderstanding how this works. All of what you are saying is supported by composing backends. Backends ultimately decide the performance cost, not the picking plugin. PointerHits need to be a unified stream - that is what drives pointer events for any entity, it has nothing to do with the performance cost of raycasting. In normal use, you will usually see 0-1 of these events per frame per pointer.

I think a core issue here is that we're computing mesh intersections across all spawned meshes by default.

This is a function of the mesh picking backend settings. Downstream, bevy_mod_picking provides this type of backend for debug purposes, but you can also use components to instead opt-in to picking where you need it. Again, this raycasting backend is essentially for debugging. It's fast enough that it's fine for most scenes, but, as I've said before, bevy's default should probably be a physics or shader based backend. We aren't there yet though, and this provides a way to get things working for 3d cases right now. The point of this architecture is that it is completely modular, and users can choose whatever picking backends make sense for their use case. Users are free to disable this backend, and drop in one of the existing 3rd party avian or rapier backends.

The "pick any mesh by default" behavior really only makes sense in editor scenarios, which is not what our defaults should be optimizing for.

Agreed.

Physics collision shapes are generally used [..]

These implementation details can exist in the backends, specific to their implementation. For example, the various physics engine backends that mod_picking provides out of the box do use collision shapes like this, but it is specific to each integration, and what they support (e.g. raycast early exit and component filtering).

(a global PointerHits event queue) has very limited practical applications.

A generic PointerHits (or picking interaction) event across all types isn't going to cut it for most applications, both in 2d and 3d.

I completely disagree. This is what allows us to support rich picking events for any entity and across picking backends without prescribing how the hit tests are run. There is no cost to this - the raycasting backend behavior has nothing to do with the unified pointer hit stream.

[aevyrie]

To respond to the issue at hand:

1. This is a bug in the mesh picking backend, we shouldn't be emitting errors, but we can document what kinds of meshes the backend supports. I don't see any reason why we couldn't support other topologies.
2. The mesh picking backend should be off by default. Adding it when you need it should be a one liner add_plugin(). Keeping the rest of the picking plugins active has negligible performance cost - if there are no hit events to consume, there is no work to do.

[cart]

@NthTensor

it's basically just intended for use in the examples (where it's very handy) and the editor prototypes (where it's basically required for a bunch of core interactions).

Yeah this is the space I think the current mesh backend is right for. Glad it seems we're all in agreement on this.

Just so we're all on the same page, PointerHits aren't really something users are supposed to consume. These events feed the HoverMap which tracks pointer hover state, and then this drives the Pointer events (see the pointer_events system for more information).

Fundamentally this is all still the same "feed" though in terms of signals being sent out.

Following your example, if you want to attach some on-click functionality to all player colliders, you can do something like the following.

This has a number of downsides. In the current impl, it implies that we're computing all intersections for all meshes ahead of time, computing a single global click result, and then filtering down to PlayerCollider.

@aevyrie

I think you might be fundamentally misunderstanding how this works

Pretty sure I'm not, but I'll keep an open mind. If after reading this message you still think I'm missing something, please let me know.

Backends ultimately decide the performance cost, not the picking plugin

I understood this much.

PointerHits need to be a unified stream - that is what drives pointer events for any entity

This is (one of) my fundamental sticking points (the other being the much more pressing concern of the current mesh picking backend not being a suitable default, which it sounds like we all agree on). I'm asserting that a single unified "pointer event" (and how these relate to each other, such as blocking hover events across entities and types of entities) isn't fine enough granularity for many use cases, especially in 3D. It makes presumptions:

Hover (Over / Out) requirements are tied to click (Down or Up) requirements when it comes to "blocking". This makes sense in UI and editor scenarios, where generally the only question you need to answer is "is this interactable" (if yes, enable is_hoverable) and "does it block when determining other hovers or clicks" (if yes, enable should_block_lower). In game scenarios, this is much more fluid. You could easily want to have hover information for something your mouse is over, that is not "clickable" where the hover observer triggers when it is behind something that is clickable (and blocks other clickables).

The system also assumes that the "click stop" information is fixed. Real world apps will have more complicated use cases that fall outside the "ui pointer model". Ex: "click the item under my mouse with the highest rarity". Or based on the state computed this frame, the player might be in "item selection mode", "enemy selection mode", or some combination of the two. This context could be determined based on what is under the mouse currently.

A very solid chunk of picking scenarios are best expressed as "describe what intersections you want to check", "retrieve the list of matches", and "from that list select zero or more entities".

I'm definitely not saying that the picking system as implemented today is not useful or that it needs to fundamentally change. bevy_picking solves very real problems and I love that it is pluggable. I'm just saying it isn't a complete answer, and that a huge chunk of the "mesh picking" scenarios in 3D (but also sprite picking in 2D) will not be covered by a global "2d-ui-style" hover/click system (at least in its current form).

My one primary goal for this discussion was to convince people that the current mesh picking backend should not be enabled by default (which seems to have been successful). My secondary goals are to (a) convince people that bevy_picking in its current form is not a one-size-fits-all solution for picking (I suspect this won't be hard or contentious), and (b) get people thinking about ways to provide some of the UX benefits of bevy_picking (observers, hover event, etc) in the context of more diverse pointer-picking and raycast operations. I suspect that there is potential for unification or capability-sharing between bevy_picking and the more diverse app-logic-driven raycasting/picking scenarios. But maybe its best to treat them as fully separate implementations!

I apologize for my phrasing in these cases: "A generic PointerHits (or picking interaction) event across all types isn't going to cut it for most applications" and "I'll assert that the current picking behavior (a global PointerHits event queue) has very limited practical applications". I could have phrased this (more accurately) in a way that didn't minimize the very real / common / valuable scenarios covered by bevy_picking.

I also think that a bevy_picking backend fed by user-provided colliders is practical, within the current bevy_picking paradigm. But again, it doesn't feel like it even closely resembles a complete solve to the picking problem. It will work for simple things (which is a large set of things!), and then when the user's / app's needs expand, they'll need to move to a different system.

This is a bug in the mesh picking backend, we shouldn't be emitting errors, but we can document what kinds of meshes the backend supports. I don't see any reason why we couldn't support other topologies.

Agreed. Not being able to pick something doesn't feel like an error to me, unless the developer has explicitly asked for it to be pickable.

The mesh picking backend should be off by default. Adding it when you need it should be a one liner add_plugin(). Keeping the rest of the picking plugins active has negligible performance cost - if there are no hit events to consume, there is no work to do.

Agreed. UI picking makes perfect sense / should of course be enabled by default. Sprite picking falls into a similar category, although 2D games are where I think we start to see the limits of the current unified picking system.

[Zennii]

I'd like to throw my own experience with it in as well, as I hit something similar the other day when updating. I'm using custom attributes with my meshes, so for example I'm packing vertex positions into a u32. This lead to massive spam of Mesh does not contain vertex positions, because the picking system looks for the common way to store vertex positions. I don't really have a problem with that, but I was surprised that it was not so easy to disable bevy_picking. Here's the order I considered my options. My initial instinct was that I could change necessary settings through .set(PickingPlugin { ... }) on DefaultPlugins similar to how the WindowPlugin and some others work, but that didn't seem to be the case. I didn't want to manually insert all the default plugins except the PickingPlugin so I skipped that idea. So I tried to just disable the feature flag, but I realized it's tied to the UI too and I probably want that, so that wasn't an option. The only way I found to solve the problem was to have a Startup that modified the MeshPickingBackendSettings to set require_markers to true (IMO true as default kind of makes more sense), and that's what I went with, but it doesn't seem like it should be necessary to need a Startup tied into this process

[aevyrie]

@cart

A very solid chunk of picking scenarios are best expressed as "describe what intersections you want to check", "retrieve the list of matches", and "from that list select zero or more entities".

You are describing the existing raycasting API (and those of rapier/avian) verbatim. My point is that this has nothing to do with picking. Please bear with me.

You can invent endlessly complex hypothetical scenarios, but you always have the option of writing a backend that has the performance and layer traversal characteristics you want for your game. If your use case truly does not fit into the paradigm of UI pointer interactions (which seems incredibly unlikely to me), then... don't use the UI pointer interactions. You still have the full ECS toolkit at your disposal.

Finding out which thing is directly under the pointer is not as trivial as raycasting into the scene. You can have multiple cameras being layered with UI and each other. The entities on screen are often rendered in completely different ways: UI, 3d meshes, 2d meshes, gizmos, custom renderers. You need to be able to have multiple sources of hit tests that can interleave properly to match the order of what is on screen. That is the hard problem that "picking" solves.

You could easily want to have hover information for something your mouse is over, that is not "clickable" where the hover observer triggers when it is behind something that is clickable (and blocks other clickables).

This is what backends are intended for! Just because the picking spec says you should report the top hit(s) and send event(s), that doesn't preclude you from bolting on your own custom logic for entities behind the hoverable ones. You can trigger those hover handlers manually, skipping the built-in focus logic entirely. The focus system is just one way to trigger the pointer events, it (shouldn't) own them.

The system also assumes that the "click stop" information is fixed. Real world apps will have more complicated use cases that fall outside the "ui pointer model". Ex: "click the item under my mouse with the highest rarity". Or based on the state computed this frame, the player might be in "item selection mode", "enemy selection mode", or some combination of the two. This context could be determined based on what is under the mouse currently.

I just don't see what this has to do with picking. If what you are doing falls outside of the UI pointer model, and all you want to do is hit tests, then do hit tests however you want! I don't see why you can't run arbitrary hit tests and trigger click handlers yourself.

The focus system is the entire point of entities participating in the combined stream of hits (i.e. "picking"). If you don't want the focus system, you can still use picking functionality that lives above (pointer events) and below (pointer locations, hit tests) the focus system. This combined focus system is something that I really don't want to lose.

I do want to add that I've been repeatedly encouraging others to make a better frontend (pointer events). @NthTensor has done a great job porting the mod_picking version of pointer events to observers, but I'm not particularly attached to it. The frontend was an early attempt at ECS callbacks that predates bevy's observers by more than a year, iirc.

I'd also support improvements to the focus system, but I want to be sure we don't end up losing what is, I think, the most valuable part of the picking plugin's current design.

[NthTensor]

I think I understand now: My sense is that we are still talking past each-other a little bit. You are coming at this from a much more user-focused prospective than we are (which is good imo) but it also seems like maybe you have not actually tried building the abstractions you are talking about using the new system. I say this, because I think the new system is pretty great for what you are talking about, even if it's only one of the necessary building blocks.

This has a number of downsides. In the current impl, it implies that we're computing all intersections for all meshes ahead of time, computing a single global click result, and then filtering down to PlayerCollider.

It's probably worth noting that we are not computing a global result, as I think you are aware. A pointer can end up hovering multiple entities, and PickingBehavior allows you to configure if a certain entity "blocks" lower hits or not, and you can also opt-in or opt-out of hovering.

Hover (Over / Out) requirements are tied to click (Down or Up) requirements when it comes to "blocking". This makes sense in UI and editor scenarios, where generally the only question you need to answer is "is this interactable" (if yes, enable is_hoverable) and "does it block when determining other hovers or clicks" (if yes, enable should_block_lower). In game scenarios, this is much more fluid. You could easily want to have hover information for something your mouse is over, that is not "clickable" where the hover observer triggers when it is behind something that is clickable (and blocks other clickables).

This is true within the picking system but you can also just turn off blocking entirely and figure out what interrupts what in user code. It's just an optimization for when these are tied, which is often.

The system also assumes that the "click stop" information is fixed. Real world apps will have more complicated use cases that fall outside the "ui pointer model". Ex: "click the item under my mouse with the highest rarity". Or based on the state computed this frame, the player might be in "item selection mode", "enemy selection mode", or some combination of the two. This context could be determined based on what is under the mouse currently.

I see, and I don't disagree. But the feedback I've gotten from the few early adopters is that they are pretty painlessly building their own custom "selectors" on top of the bevy picking observer events. Architect of ruin has implemented a heuristics based selection system using it, by combining observers and some additional event streams.

You seem to be saying that this functionality should be easier to build, or provided in the engine core. That makes sense, and I think the existing picking HoverMap is probably still, on the lowest level, the right tool for the job. Are we talking about making the picking front-end smarter and more modular? I think that would be great.

I'm just saying it isn't a complete answer, and that a huge chunk of the "mesh picking" scenarios in 3D (but also sprite picking in 2D) will not be covered by a global "2d-ui-style" hover/click system (at least in its current form).

I'm tentatively with you. I think picking is quite flexible, and the feedback i've gotten from users so far is that they are building more complex interactions on top of picking rather than along side it.

My one primary goal for this discussion was to convince people that the current mesh picking backend should not be enabled by default (which seems to have been successful). My secondary goals are to (a) convince people that bevy_picking in its current form is not a one-size-fits-all solution for picking (I suspect this won't be hard or contentious), and (b) get people thinking about ways to provide some of the UX benefits of bevy_picking (observers, hover event, etc) in the context of more diverse pointer-picking and raycast operations.

Yep. I'm with you for both of these, I think. But @aevyrie and I are asking about scope and relevancy: how much of this is really "picking" and how much is a more general question about bevy's preferred approach to user interaction behavior?

---

I really want to make sure we're on the same page here, so here's a little description of what bevy_picking actually is for. Sorry if this is repeating information you already know.

The fundamental task of bevy_picking is to take points on windows, images, and textures and relate those positions to the entities "visually beneath" those points. This is very simple: back-ends turn PointerLocations into PointerHits events for every entity they hit.

The second tasks of bevy_picking is to drive the positions of those points by mouse/touch-pointer/stylus input events, and expose the input interactions to the entities beneath those pointers via dispatching Pointer<E> triggers.

Connecting these two modules together is the HoverMap, which tracks which pointer is "hovering" which entities, and which can be used to build more complex interactions that are still driven by raw pointer input.

It seems to me that you are suggesting a third task, and it's still not entirely clear to me what it is.

[cart]

@aevyrie

You are describing the existing raycasting API
Finding out which thing is directly under the pointer is not as trivial as raycasting into the scene. That is the hard problem that "picking" solves.
The focus system is the entire point of entities participating in the combined stream of hits

I know. Please give me the benefit of the doubt here. I've used raycasting APIs extensively, both in the context of Bevy and elsewhere. I also know what picking is, as you have defined it (which is aligned with how pretty much everyone defines it). I know why bevy_picking exists, and I think its existence is well motivated and justified. I like the current implementation (which I have reviewed extensively). We don't need to argue about this, as there is nothing to argue about.

One potential hangup here seems to be whether we regard "picking" as "asking questions about what is under my mouse and returning those results" or "finding what is directly under the pointer, solving the specific 'exposing hits across sources, sorting them, and picking the closest' problem". I'm happy to accept that definition and call the scenarios I've described something else. I believe they exist somewhere on the spectrum between "arbitrary raycast scenarios" and "picking a depth sorted result under my mouse, taking blocking into account", in such a way that we might be able to share infrastructure in places (ex: over/out hover tracking, events, etc). But for clarity and to avoid further controversy, I will no longer refer to the more general scenarios as "picking".

You can trigger those hover handlers manually, skipping the built-in focus logic entirely. The focus system is just one way to trigger the pointer events, it (shouldn't) own them.

This is the category of thing I was trying to touch on (sharing infrastructure with the more general "asking questions about what is/was under my mouse" scenarios). This is the "3rd scenario" as @NthTensor described it, which is what I was trying to describe above.

@NthTensor

It's probably worth noting that we are not computing a global result, as I think you are aware. A pointer can end up hovering multiple entities, and PickingBehavior allows you to configure if a certain entity "blocks" lower hits or not, and you can also opt-in or opt-out of hovering.

The "global" result I'm talking about is the "depth sort and block" result, which determines if hover and click events are fired. The choices made by an entity above you (provided by an arbitrary backend) affect whether your hover or click events fire. There is (currently) no way to say "I want hover events no matter what is above me" for a specific entity. The system assumes there is one question to answer each frame (what is being "picked" as defined by bevy_picking, accounting for blocking configuration). If you need to ask multiple questions with multiple constraints in a given frame, or if you need to ask a question that the current system doesn't allow you to ask, you are on your own.

Again, I fully acknowledge that this is a Good Thing for picking. We want that. But with a singular focus on answering the "picking question" we have prevented a category of other questions from being asked. HoverMap cannot be used to answer these questions, because it is also informed by that global sort and the general picking system constraints (again ... picking is a good thing and the way it is implemented is good).

My "3rd scenario" is "when people want to ask questions about what is under their mouse (and what was under their mouse) that the global picking sort behavior and individual entity configuration cannot answer (edit: or that a single configuration of that system cannot answer within a given frame) , how can we make that as efficient and ergonomic as possible, and potentially reuse infrastructure that has already been built". Some various (not necessarily related) ideas:

1. bevy_picking is already getting fed raw (unfiltered) hits from backends via PointerHits events. This is a great design that enables us to build on top of it. When we have a "new" question to ask, rather than recompute this data again via new raycasts, we could instead consume that event feed and then filter it down based on arbitrary criteria (and perhaps filter down to only the backends or entities we want to consider).
2. bevy_picking computes a (private / Local) Overmap from the PointerHits events. To my knowledge this contains everything necessary to compute arbitrary over/out events (including new arbitrary derived "filtered" over/out results with different criteria, in the style of the current HoverMap). We could discuss making this (or something like it) public.
3. We could consider generalizing the PointerHits -> Overmap -> SOME_FILTERING_PROCESS -> Event Triggers system in such a way that allows someone to define some high level PointerQueryPlugin::<T: PointerQuery>::new() that would generate over/out (and perhaps down/up) event triggers based on some PointerQuery impl. Where T could be something like "filter to entities with Rarity, sort by Rarity, and select the 3 rarest entities". The user just defines the filters and selectors and the query does the remaining tracking stuff. In/out events would be fired based on what has just become one of the 3 rarest entities under the cursor (or what has just stopped being one of those entities).
4. We could consider doing something like (3), but generalized to "arbitrary raycasts", not "cursor raycasts".

I hope that helps convey where my head is at / illustrates that overlap + generalization is possible. Again this was all a secondary concern relative to disabling the current mesh picking backend by default. We don't have to engage with these ideas further if they are uninteresting to you. Certainly not a high priority right now.

[aevyrie]

@cart I'd like to continue discussing this, but maybe we should start a new issue? I'm worried we are OT for this particular issue, and I've opened a PR that would close this if merged.

Edit: I'll create one.

[NthTensor]

Ok, now I understand and I think I'm totally on board. Thanks for explaining, sorry about the confusion.

We can continue that off-shoot in the other thread. For now, I totally support your more direct suggestions for the mesh picking backend.