This project demonstrates sonosthesia unity packages which allow XR interaction with procedural graphics and sound. These interactions combine a number of different techniques:
- physics collisions and triggers (using Unity Physics)
- hand shape analysis and pose detection (using XR Hands)
- continuous data flow and processing based on Rx
- a custom affordance system based on interaction sources and actors
Fist, open and pinch hand shapes have different effects on the reactive assets.
XR controller interaction is implemented as mimicking hand shape through a combination of grip and trigger. It is mostly used for debug porposes, more sophisticated controller interactions will be implemented in later iterations.
These demo scenes include reactive sound, videos including sound are available. System menu pinch gestures are available to bring up scene selection UI (left hand) and recenter scene (right hand).
- You must open and build
FMODProjectin FMOD studio - Android build settings are for Quest 3 (you can change the
Oculus/OculusProjectConfigasset for aditional device support) - Standalone settings can build Link executable on Windows but hands are not supported.
- Do not apply the removing HDR support suggestion as it will remove bloom effect.
Video with reactive sound available here.
Video with reactive sound available here.
This project illustrates the use of the following sonosthesia Unity packages.