Animated meshes disapear if their original position is not within the field of view

[nicopap]

Bevy version

cdb62af (main as of 2022-06-09)

What you did

I've a giraffe where the eyes are a different mesh from the rest of the body, and when the neck moves enough, it's possible that the eyes after application of animations are so far away from their initial position that the engine thinks it's outside of frustum, while they should be within view, and doesn't render them.

Video demonstrating the bug:

giraffe-model-disapear.mp4

As you can observe, the blue eyes disappear if the original eye position are not within the field of view.

The rest of the mesh (skin) also disappear when the camera view doesn't include the original mesh's position.

Additional information

This is because entity mesh AABBs are not computed based on the animation. The animation shader runs on the GPUs, while AABB filtering runs on CPU before anything is sent to the GPU. There is no way for bevy to filter meshes based on the animation position.

This seems to be something barely possible if not impossible to fix within bevy. Suggestions include:

• Pre-processing step for generating AABB before loading models, accounting for joint limits or pre-backed animations.
• Cunningly associate the invert-bind-pose to the vertices it's supposed to transform, create AABBs for each bones, and use that AABB with the transform of the bone for culling of animated meshes. (note that it's not a full solution and it's fairly expensive). This is a rough idea and I'm not sure if it can even work.

Workaround

There are two options if you hit this issue:

• Add the NoFrustrumCulling component to the entities which mesh disappear because of animation.
• Change the Aabb manually so that it can cover the full range of possible visible position for the model. (note that this workaround might break when [Merged by Bors] - Recalculate entity aabbs when meshes change #4944 lands)

[aevyrie]

A simple starting point might be to use a conservative bounding sphere of the longest bone chain to generate an AABB:
https://gamedev.stackexchange.com/questions/43986/calculate-an-aabb-for-bone-animated-model

[Shatur]

This also causes raycast to work incorrectly for animated meshes (because actual mesh could be in a different location).
@aevyrie is it something that should be addressed in raycast implementation or could be fixed on Bevy side?

[nicopap]

The problem with raycasting is double:

• raycasting uses AABB to narrow down which mesh to look at, if the AABB is bogus, well it won't pick up the right meshes when narrowing down
• raycast uses the bevy Mesh, which is the CPU side representation of the model, the CPU-side representation has no awareness of the position of vertices when animated, so it will always be wrong

If you content yourself with CPU-side raycasting, you'll have to accept either ridiculous performance loss or inaccurate (too broad) collision detection.

But there is a way! This is where GPU picking is useful. GPU picking constructs a buffer with individual ID per entity, you can then read that buffer for the pixel under the cursor, and you have the picked entity!

Issue is I don't think someone has yet made a GPU picking implementation in bevy.

[torsteingrindvik]

As a note, I had a GPU picking impl that worked with skinned meshes as well a while ago: #6991

it is superseded by @IceSentry 's PR here: #8784

I see the most recent comment there by @mtsr is to consider having GPU picking as a pre-pass.
I don't know how/when/where culling happens so I'm not sure how relevant it is for this issue.

[coreh]

Workaround snippet, for convenience:

use bevy::{
    prelude::*,
    render::{mesh::skinning::SkinnedMesh, view::NoFrustumCulling},
};

/// System that automatically disables frustum culling for
/// all skinned meshes, as soon as they are added to the world.
fn disable_culling_for_skinned_meshes(
    mut commands: Commands,
    skinned: Query<Entity, Added<SkinnedMesh>>,
) {
    for entity in &skinned {
        commands.entity(entity).insert(NoFrustumCulling);
    }
}

Then just add it as a system:

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Update, disable_culling_for_skinned_meshes);
}

[Sirmadeira]

This issue doesn't occur solely with animations. It occurs with any type of entity that has a skeleton animation. In my case scenario, I have a player following camera. And if i stop looking at the mesh boom, no rendering. Or shift the orbit in y boom half rendering.

[Preston-Harrison]

This took me a while to figure out today. Until a proper solution is landed on, it would be nice to disable frustum culling for skinned meshes by default. This seems like it would affect almost every 3d game, and can be a bit of a head scratcher.

[greeble-dev]

I've made a standalone plugin that mostly works around this issue but has some drawbacks.

I think the plugin could be the foundation of an engine fix, but my path to a PR is unclear. I'm going to lay out some notes below and see if there's any feedback.

Summary

The plugin finds entities with skinned meshes, adds a new component+asset with per-joint AABBs, then uses them to re-calculate the mesh's AABB every frame. This approach is robust - as in the mesh is guaranteed to be within the AABB - as long as the mesh's vertices are only driven by linear blend skinning (no blend shapes or vertex shader shenanigans).

skinned_aabb.mp4

So, why not just add this plugin to the engine's default plugins? There's two key issues:

1. Concerns over per-frame performance.
2. The per-joint AABBs should be pre-calculated at some point in the asset pipeline, but I'm unsure what the solution would be.

There's a few other smaller concerns but I'll skip them for now.

Issue 1: Per-Frame Performance

I've benchmarked the plugin on many_foxes, and my hand-wavey conclusion is "enabling skinned AABBs multiplies the main app CPU cost of a mesh playing a single animation by 1.04x". Alternatively, "on a mid-range desktop CPU, each skinned joint costs an extra 8ns". More detailed notes are here.

Is 1.04x is a fair price for fixing a common problem users have with skinned meshes? I'd say yes.

There's also ways to optimise:

1. Give users the option to choose cheaper but less robust approximations.
2. Improve the robust solution by trying different shapes, compression, avoiding global<->entity space conversions, or combining with other systems that process joint transforms (animation, render extraction).
3. As the renderer gets more GPU-driven the skinned AABBs might move to the GPU and be calculated during frustum culling.

Also worth noting that some people are working around the original issue by disabling frustum culling. This increases renderer and GPU costs, so they might find switching to skinned AABBs is a performance win.

Issue 2: Asset Pipeline

The plugin calculates the per-joint AABBs from the mesh vertices at runtime - also requiring the mesh vertex data to be kept in the main world asset - and has to continually check for new meshes. A real fix should allow all this to be done at asset import time.

In the long-term my guess is that the Bevy mesh pipeline will become more modular and extensible, so adding a skinned AABB calculation in the right place will be easy.

But in the short-term, my guess is that most Bevy projects are instantiating scenes directly from GLTF files. We'd need a temporary solution that modifies the GLTF importer.

Solution 1: Add a GLTF extension to store the skinned AABB data and create a tool for users to pre-process their GLTFs. Annoying for users and the solution is likely to be thrown away at some point.

Solution 2: Extend the GLTF importer to calculate the skinned AABB data while loading. Has a small load time performance cost and will require some moderate refactoring of the importer. But is simple for users, and the refactoring might align with other proposals to modularise the pipeline (#13681). Eventually the code would migrate out of the GLTF importer and into a pipeline module.

What Do Other Engines Do?

Most game engines have a "fixed size AABB attached to the root bone" path for skinned meshes. Some have approximations that look at other joints, but none are robust.

• Unreal: Root bone bounds are automatic (from reference pose) or custom. Also supports manually authored per-joint bounds via physics assets.
• Unity: Root bone bounds are automatic (from animations) or custom (documentation).
• Godot: Unclear. There's an issue that says they only support root bone bounds (AABB broken on imported meshes with skeletons. godotengine/godot#85171), but also code in the engine related to per-joint AABBs (MeshStorage::mesh_get_aabb). I don't know if they're using this at runtime.
• O3DE: Root bone bounds, plus option to approximate the AABB from joint origins and a fudge factor.

What Next?

Right now I'm considering two paths:

1. Forget about engine changes in the short-term. Continue building out the standalone plugin.
2. Or, start investigating the GLTF importer refactoring and see how far I get.

If there's positive vibes around changing the GLTF importer then I'd likely choose that path.