Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make MediaPipeController tracking local to model #63

Closed
wants to merge 1 commit into from

Conversation

copygirl
Copy link

@copygirl copygirl commented Dec 10, 2024

Fixes #59.

Removes the use of global_transform in MediaPipeController for tracking purposes. This made it impossible to position the ModelController in any way without also affecting the tracking by tilting the model, and making the hand offset static to a certain location in-world.

Some changes to note:

  • I parent the head and hand nodes to the model, just like the rest hand reference ones are.
  • Fixes my model floating 0.4m above the ground because of a hardcoded model offset.
  • Since the head tracking values are relative to the camera, I figured we don't NEED head_vertical_offset any longer (I set the default to 0.0) if we just make it relative to the default resting head pose.

Notable issues that I can remember that might need to be addressed:

  • Changes to IK are likely incorrect.
  • Relatedly, ducking my head results in my legs tiling forward.
  • Even with head centered on the camera, it's not centered, causing leaning to the side.
  • The code that previously normalized the model's location so the head was up-right is broken.
  • Hand landmarks had to be made top_level to display correctly, thus do not move with the model.
  • Commented out some code which didn't appear to have an effect. What it do?
  • Raw hand tracking data appears to not be relative to the camera like the head data is. I ended up hardcoding a 15cm offset to make it line up properly. Why is this?
  • See if VMCController needs a similar treatment.
  • The hardcoded camera_origin_offset could be replaced with a calibration method.
  • A head transform calibration would also allow for the camera to not have to be placed directly in front.

@copygirl
Copy link
Author

Kiri has said there were some upcoming changes that will likely make this PR obsolete (or just hard to fix all of the merge conflicts), and in the meantime I've instead put my efforts into refactoring / rewriting from scratch a lot of the tracking code, both to get a better understanding and maybe help hunt down some bugs of which I appear to have found some already. That work can be seen here. There's still a number of things to reimplement to have feature-parity but I'm happy to share this work with SnekStudio.

For now I think this PR can be closed, considering I won't consider supporting it.

@copygirl copygirl closed this Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Don't assume model is positioned at world root
1 participant