Add an option to apply blendshapes and generate a MeshInstance #24
Comments
Lexpartizan
changed the title
|
I'm having a hard time understanding your request.
|
|
I'm having a hard time understanding your request.
|
|
If this could be solved with just a few lines of code, why not implement such a function into one of the mesh classes. This would not effect performance at all for those who don't use it. |
|
So you can give me this few lines of code? |
|
I think that the engine is already building the surface I need (How else would a 3D model be displayed from a blendshapes?), but I need a function so that I can get the resulting mesh. |
I am going by the idea that this could be solved by a script. Sorry, I don't actually know how to do it. Good luck though, keep researching |
|
A possible design could be this:
|
|
my "research" did not lead to anything. Not my level programming. |
|
Merge all the target blend shapes to one blend shape How make this in godot? But I don’t know how to do this in a godot with code. |
|
This is for the one blend shape and will get the blend shape mesh. Once you have the blend mesh and the original mesh, you can read the vertexes and linear interpolate with the blend value between 0.0 and 1.0. https://docs.godotengine.org/en/3.1/classes/class_@gdscript.html#class-gdscript-method-lerp Break up the three components of the position. Use lerp ( target_mesh_float, blend_shape_float, blend_weight ) over all the vertex locations. Edited: |
fire
changed the title
|
Thank you, you gave me something to think about. Yeah, that makes sense. Although I still don't understand how to work with vertices. Not my level. So help is still needed. |
|
Well, it is possible to do just in plain gdscript. See my repository at https://github.com/gamedev-kindness/make-target What you generally do
The problem is that there is no way to modify ArrayMesh vertices en masse with sparce indices - i.e. supplying array of indices and array of deltas and modify mesh in one shot in most effective way. I think this is what mostly needed for the solution to succeed. |
|
@slapin |
|
You do not need to update every frame, you just update your vertices once
and then work with the result. The texture is just good way to store deltas.
See SIMS4 GDC video about their editor for more in-depth explanation.
…On Sat, Sep 7, 2019 at 11:46 AM Lexpartizan ***@***.***> wrote:
@slapin <https://github.com/slapin>
I don't really understand how to generate UV texture and how to work with
colors in it. Also, I think the offset and vertex recalculation of each
frame in the viewport will have a negative impact on performance. I will
try a couple of weeks, looking at Your code, start working with vertices
and get some result on my own. But I will definitely address you on discord
(especially since You speak Russian). I understand You're working on a
similar project.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24?email_source=notifications&email_token=AAABPUZCELEYRSLVLUIIZXTQINS5FA5CNFSM4ITZKWZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ET3RI#issuecomment-529087941>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAABPU5XXGZ3IESCHP4LTG3QINS5FANCNFSM4ITZKWZQ>
.
|
|
This proposal explains technical dependency to make proper character customization editors: #41 |
|
I have some progress witn that. |
Are you aware of this project: |
|
@golddotasksquestions, |
|
The work was done great, especially artistic side. However there are some
shortcomings with this approach:
1. Use of blend shapes prevents it from working with Godot GLES2 renderer.
2. Use of non-uniform bone scale breaks animations
More advanced examples of this approach is Honey Select rig and Unity UMA
rig - they use dedicated bones for non-uniform scaling.
Also, having transforms constantly set might affect performance for larger
skeletons, so keeping transform on many bones (via set_custom_pose)
might be not a good idea if you want many characters. And I seen that games
using this approach usually have 2-3 characters on screen, no more,
so I guess that is quite expensive. Sims4 approach is much less expensive
as characters end up optimal, without extra bones, without (unnecessary)
blend shapes.
However, if your only platform is PC or current gen consoles, you can use
editor like and then process the result and optimize it as additional pass.
You will need to apply bone scale, apply blend shapes to mesh and remove
blend shapes. Also you will need to pack materials.
…On Sun, Sep 8, 2019 at 10:33 PM Lexpartizan ***@***.***> wrote:
@golddotasksquestions <https://github.com/golddotasksquestions>,
No, thanks for information. Good generator! Download and watch his code
tomorrow.
I trying just make Makehuman for Godot.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24?email_source=notifications&email_token=AAABPU4JCIFEZBGYLW4F5ETQIVHRDA5CNFSM4ITZKWZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6FXTWA#issuecomment-529234392>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAABPU2UDSETPTQ327756WLQIVHRDANCNFSM4ITZKWZQ>
.
|
|
Is there any way to get ArrayMesh with a regular array with vertices, UV coordinates and so on? From them you can build ArrayMesh, but I would just like to read these values. I use MeshDataTool now, but it is too slow. I would like to get a regular array of vertices and work with them. And then create a new ArrayMesh. Now i have 50 different characters on the screen and 60 frames per second. Video Memory consumption= 35 MB. But I want a faster way to create character. Sorry. I already found surface_get_arrays ( int surf_idx ) Thanks! |
|
See ArrayMesh doc, I write from memory:
surface_get_arrays()
surface_get_blend_shape_arrays() (IIRC)
You don't need meshes to be a part of MeshInstance for this.
…On Mon, Sep 9, 2019 at 8:12 PM Lexpartizan ***@***.***> wrote:
Is there any way to get ArrayMesh with a regular array with vertices, UV
coordinates and so on? From them you can build ArrayMesh, but I would just
like to read these values. I use MeshDataTool now, but it is too slow. I
would like to get a regular array of vertices and work with them. And then
create a new ArrayMesh. Now i have 50 different characters on the screen
and 60 frames per second. Video Memory consumption= 35 MB. But I want a
faster way to create character.
In addition, there is one small bug. When I use the mesh in the scene, but
he's laying there just as the data (I read blendshapes from this) and the
screen is not displayed, it still consumes the VideoMemory (VideoMemory,
not RAM!), so I should just reset this value.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24?email_source=notifications&email_token=AAABPU3DT54MILXVUGNI4XLQIZ7XPA5CNFSM4ITZKWZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ILN2A#issuecomment-529577704>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAABPUY3C5QN2ZJRT7XC5X3QIZ7XPANCNFSM4ITZKWZQ>
.
|
|
Yes, i dont use meshes. |
|
So slow! 0.5 sec for model with 16000 triangles. |
|
do you mean generator part or display part?
Generator part is OK to be slow as it is done once per asset.
Anyway, I work on C++ version of generator now, and it is from 100ms to
1sec for character to generate raw shape data without compression
which is basically triangle rasterization on CPU. I think 100ms will be max
time to apply shape using C++ version.
Godot developers said big NO to all ways to optimize this so only way is
engine fork to add this feature.
…On Tue, Sep 10, 2019 at 9:36 PM Lexpartizan ***@***.***> wrote:
So slow!
for i in range(size_vertex_arr):
basis_mesh[Mesh.ARRAY_VERTEX][i].x=lerp
(basis_mesh[Mesh.ARRAY_VERTEX][i].x, blendshp["blndshp"][name_shp][0][i].x,
value)
basis_mesh[Mesh.ARRAY_VERTEX][i].y=lerp
(basis_mesh[Mesh.ARRAY_VERTEX][i].y, blendshp["blndshp"][name_shp][0][i].y,
value)
basis_mesh[Mesh.ARRAY_VERTEX][i].z=lerp
(basis_mesh[Mesh.ARRAY_VERTEX][i].z, blendshp["blndshp"][name_shp][0][i].z,
value)
0.5 sec
if 200 blendshapes - 100 sec for each character...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24?email_source=notifications&email_token=AAABPU4MA44KBTS3TDW7OBLQI7SMFA5CNFSM4ITZKWZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6MCVEI#issuecomment-530066065>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAABPU3WJ4F6OEFLUTF3YHDQI7SMFANCNFSM4ITZKWZQ>
.
|
|
It is very sad. It is interesting how surface with blendshapes are rendered. Better, as I understand it, the engine builds the surface we need with the help of the GPU. And then it displays. It remains just in time to get this surface ... Yes, C ++ will allow you to use this with GLES 2, but it doesn’t interest me much. |
|
My mesh is 26K triangles, 76 blend shapes. Code is not quite optimized
though. 100ms is for whole mesh.
But do not consider it a final number, it is mid-development.
…On Tue, Sep 10, 2019 at 10:26 PM Lexpartizan ***@***.***> wrote:
It is very sad. It is interesting how surface with blendshapes are rendered. Better, as I understand it, the engine builds the surface we need with the help of the GPU. And then it displays. It remains just in time to get this surface ...
In theory, the work has already been done, it remains to find out how to get the result.
Yes, C ++ will allow you to use this with GLES 2, but it doesn’t interest me much.
100 ms for how many vertices and blends?
GDscript 400-500 ms for 1 blendshape and 16000 verts.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@Lexpartizan @slapin I do not plan to hijack this discussion, but I need feedback! Recently Aviv shows the pipeline from Character Creator 3 to Godot. which preserve the BLENDSHAPEs in godot. I seek feedback on
Generally, I am interested on blendshape, skin realism for making realistic 3D character in godot. |
|
@GeorgeS2019 First, character studio is paid and expensive. That's why I don't want to get involved with him. Secondly, those blendshapes that are available in the godot, as a result, allow only to control the facial expressions of the character, but not to change his face or body. That is, it is not suitable for an in-game character generator, you get only one character, even if it is of high quality. Yes, i want to realistic skin too. |
|
@Lexpartizan @fire YES! It is time the Godot community gathers interest to support in-game character generator with realistic 3D human with believable skin and emotion intense facial animation . Impressive work has progressed in 3D avartar e.g. VRM The Character Creator 3 to Godot (Tutorial) shows how to preserve facial animation with way to integrate skeletal animation from Mixamo Hopefully more people join in to this discussion so we could properly formulate a Godot proposal. |
|
I wanted to reiterate that Blendshape to Skinned meshes can be done via https://github.com/electronicarts/dem-bones and is used in recent games like TLOU2 (The Last of Us 2). There is a recent GDC talk about this. |
|
To be honest, I would prefer something like a modifier in a blender called "surface deform" to realistic skin. This would help with clothing for different characters. So that the clothes have a shrinkage on the figure of the character automatically. And a good skin is made even in substance painter, using quite a standard pbr shader. And I would also really like to have lessons on compute shaders published, since I couldn't really understand what it is, but it seems to me that they are able to solve the problem with generating a small mesh from a large one with a bunch of blendshapes. While I'm doing this on the CPU. |
|
Compute shaders won't help much with generating one mesh from another. As
for baking the geometry you just need to
lerp your 3D coorinates (and only them) to each subsequent weight, but you
better prep your data so that it is not actual
mesh but just geometry data (Meshes are copied to GPU and your mesh will be
too large with all the blend shapes). The Sims4 approach
is not bad for that (see GDC video/presentation). Other approach would be
like Makehuman, where you keep your various transformations
in relation to base mesh you modify (which is master data for whole
system). For clothes there is Makehuman approach (as mentioned, using
master mesh, each vertex relates to 3 master vertices) or KC:D approach
(using shrunk/inflated versions of clothing, shrink or inflate clothing
until it fits
(i.e. no intersections with lower layer). It is for multilayered cloth.
Both are feasible with Godot and can be implemented (both in C++ though due
to
ineffective data structures in gdscript - manipulate array of floats). PoC
code can be done in gdscript for reference.
…On Thu, Aug 5, 2021 at 8:23 AM Lexpartizan ***@***.***> wrote:
To be honest, I would prefer something like a modifier in a blender called
"surface deform" to realistic skin. This would help with clothing for
different characters. So that the clothes have a shrinkage on the figure of
the character automatically. And a good skin is made even in substance
painter, using quite a standard pbr shader. And I would also really like to
have lessons on compute shaders published, since I couldn't really
understand what it is, but it seems to me that they are able to solve the
problem with generating a small mesh from a large one with a bunch of
blendshapes. While I'm doing this on the CPU.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABPUZHRLV5TCTJYU52TRTT3IN6HANCNFSM4ITZKWZQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
|
Compute shaders won't help much with generating one mesh from another. In this case, it is quite likely that you will be able to solve not only a simple problem in the form of a lerp between the basic forms, but also prepare data, that is, calculate for each vertex of the clothing which triangle of the body it corresponds to. Of course, the data preparation can be calculated in advance and does not need realtime, but the application of this data (when we are already putting clothes on the character using this data) should be quite fast (it is not necessary to do this in every frame, you can generate meshes when loading a level, for example, but it is still very slow on the CPU). |
|
You will destroy all the performance by fetching/updating anyway. You will
need to have your data well structured so
to apply simple operation on large amount of data and then fetch. But still
it looks like huge overkill for characters.
People do some interesting simulations though...
…On Fri, Aug 6, 2021 at 8:39 AM Lexpartizan ***@***.***> wrote:
*Compute shaders won't help much with generating one mesh from another.*
In fact, the blend shapes in GODOT 4 are already processed by the compute
shader. So such shaders are very good for processing data arrays.
In this case, it is quite likely that you will be able to solve not only a
simple problem in the form of a lerp between the basic forms, but also
prepare data, that is, calculate for each vertex of the clothing which
triangle of the body it corresponds to. Of course, the data preparation can
be calculated in advance and does not need realtime, but the application of
this data (when we are already putting clothes on the character using this
data) should be quite fast (it is not necessary to do this in every frame,
you can generate meshes when loading a level, for example, but it is still
very slow on the CPU).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABPU2P6FXFVRGRCSFNDDTT3NYQTANCNFSM4ITZKWZQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
|
I know I am late to the party but However, I see three things that need to be added for this proposal to be fulfilled effectively:
I'm not convinced that a GPU-based solution would be valuable enough here. Doing it CPU-side should be nearly as fast and much more convenient. The logic can be copied from the GLES2 CPU blend shapes function https://github.com/godotengine/godot/blob/8db0bd44249e9cac56cf24c7c192bc782c118638/drivers/gles2/rasterizer_scene_gles2.cpp#L1388 |
|
@clayjohn Thank you very much! Especially for specifying a link to the code. I will definitely study and try the method from the surface tool. Update. As far as I understand, create_from_blend_shape ( Mesh existing, int surface, String blend_shape ) just extracts one blendshapes, and does not apply ALL the blendshapes as we need. And not have weight of blending. |
|
Here's a sketch of an api. Array MeshInstance3D::bake_custom_blendshapes(Map<StringName, float> blends)
Array MeshInstance3D::bake_current_blend_shapes() {
Map<StringName, float> blends;
// foreach blend shape insert into blends
Array baked_mesh = bake_custom_blendshapes(blends);
return baked_mesh;
}
void MeshInstance3D::apply_current_blend_shapes() {
Array baked_mesh = bake_current_blend_shapes();
mesh = baked_mesh;
} |
|
Status: Broken from 3.x upgrade. I am working on a Dem Bones module for Godot Engine. I am not sure if it will be accepted into Godot Engine core, but I will attempt to create the blend shape to skeletal bone Godot Engine c++ module. https://github.com/V-Sekai/godot-dem-bones
|
|
Going to poke this and add it to my list. |
YuriSizov
moved this from In Discussion
to Ready for Review
in Godot Proposal Metaverse
YuriSizov
moved this from Ready for Review
to Ready for Implementation
in Godot Proposal Metaverse
|
In Godot 4, if you dont touch the skeleton or shape keys, then the model is not updated and there is zero performance loss already. |
|
Implemented by godotengine/godot#76725. |
github-project-automation
bot
moved this from Ready for Implementation
to Implemented
in Godot Proposal Metaverse
Describe the project you are working on:
I trying make in-game character generator for godot from project MakeHuman. Like Skyrim or Fallout 3-4.
https://github.com/Lexpartizan/Go_MakeHuman_dot/
https://www.youtube.com/watch?v=cPVjNh2Ki4I
I make blend-file with one model with 200+ shapekeys. Importing to Godot.
I need setup blendshapes value and customize the appearance of the character.
I can this and on my github page have godot project with this example.
Describe how this feature / enhancement will help your project:
Then applying all shapes and generate MeshInstance without blend_shapes, because file with blendshapes for each character takes up a lot of memory space.
Mesh after import to Godot with all shapekeys- 100 Mb.
But, you need make resource local for different characters.
1 character on screen - 100 Mb video-memory. This is only vertex - memory.
10 character on screen - 1Gb video-memory.
Without blendshapes - 1 character only 2,5 Mb. Zombie rush possible! And each zombie unique.
AAA-feature with one small function.
And i dont know how make that. I don't even know if this is possible.
So, this feature request.
I think the character generator is a very important part of many role-playing games and its effective implementation it just depends on whether it is possible to apply shapekeys and get ready a small version of this mesh. Inefficient it is already working and can be used in games with a small number of characters in the frame, for example in fighting games.
I can make this in blender, but for in-game character generator this must be maked via code.
I asked this question on forums, discord, Q&A, etc and did not get an answer.
Is there a reason why this should be core and not an add-on in the asset library?:
Because this is the basic function for working with the mesh. And other functions to work with blendshape already have.
If this enhancement will not be used often, can it be worked around with a few lines of script?:
That's possible. So I want to know these lines ;-).
Now I have the nerve to ask a question already on a github. Sorry for that. And sorry for my english. I hope you could understand the problems from this vague description.
The text was updated successfully, but these errors were encountered: