[AOT] Integrating Taichi AOT with Unity

[YuCrazing]

Concisely describe the proposed feature
As a potential large need of the game industry, we should give support to integrate Taichi AOT with commercial game engines. The first one could be Unity.

Describe the solution you'd like (if any)
A reasonable way to integrate Taichi AOT into Unity is via unity native plugin interface, which provides the different low-level graphics APIs (e.g. vulkan/metal/d3d11/d3d12/opengl) and provides the ability to access Unity's rendering pipeline. The NativeRenderingPlugin (NRP) repo demonstrates how to use Unity native plugin interface.

A expected usage of Taichi AOT in Unity
Considering the new features and the portability, firstly, we can explore using Taichi AOT in Unity both with vulkan backend. And for simplicity, we can just treat the Taichi AOT module as a computing module (like compute shader) at the current stage.

Based on C API for Taichi AOT, we can give a user's expected usage of Taichi AOT:

1. Get the vulkan device from Unity to initialize the Taichi AOT module.
2. Use taichi_set_runtime_context_arg_* APIs to set kernel's parameters.
3. Run kernels in the Taichi AOT module.
4. Copy data back to Unity buffer (via native buffer pointer) to render.

Detailed usage of Unity native plugin interface
When the Unity plugin loads, UnityPluginLoad(IUnityInterfaces* unityInterfaces) is called automatically. You can get a IUnityInterfaces instance here.

Then you can get different low-level interfaces via IUnityInterfaces by using:

interfaces->Get<IUnityGraphicsVulkan>(); // vulkan
// or
interfaces->Get<IUnityGraphicsD3D11>(); // d3d11
// or
interfaces->Get<IUnityGraphicsD3D12v2>(); // d3d12

Let's take vulkan as an example:

The structure of IUnityGraphicsVulkan is source code. Here you can use IUnityGraphicsVulkan::Instance() and IUnityGraphicsVulkan::AccessBuffer(...) to access Unity's vulkan device/queue/buffers.

[k-ye]

Excited to see that Taichi is moving toward this direction!

Could you also write down the concrete interfaces provided by Unity RHI pluggins? E.g. IUnityGraphicsVulkan. Thanks!

[YuCrazing]

Detailed usage of IUnityGraphicsVulkan is added.

[k-ye]

Note: when using the Vulkan resources (e.g. VkInstance, VkDevice) from Unity, or any other platform, note that we need to call the corresponding volk init methods on them. This is done around these places:

taichi/taichi/backends/vulkan/vulkan_device_creator.cpp

Line 193 in d096f15

if (!VulkanLoader::instance().init()) {

taichi/taichi/backends/vulkan/vulkan_device_creator.cpp

Line 331 in d096f15

VulkanLoader::instance().load_instance(instance_);

cc @bobcao3

[YuCrazing]

Update: Now we can run taichi implicit fem demo on unity (linux platform, vulkan backend) using unity native plugin interface.

[YuCrazing]

Implementation details:

1. Initialize taichi AOT module using unity vulkan instance at the first frame.
2. Execute computing in taichi AOT module and transfer data to unity vertex buffer every frame.

It sounds quite straightforward. However, there are still some critical problems that need to be solved next:

Taichi side:

• When using Taichi AOT C++ demo as a shared library in the NRP project, there are many vulkan header conflicts. We should avoid including vulkan headers in taichi vulkan backend header files as possible.
• Taichi AOT module should provide APIs to let users specify buffer (ndarray) size.

Unity side:

• Now taichi buffer -> unity vertex buffer data transfer happens on CPU, we need to explore to transfer data on GPU directly via unity native plugin interface.

[PENGUINLIONG]

PR #5204 introduced a C# language binding generator for unity integration, so C# scripts can directly interact with the C-APIs as a native plugin in the future. It feels to me that it solves most of the problems we have atm, including on-device copy. I'm gonna continue to work on the implicit-fem demo and see if it feels good to Unity developers.

[PENGUINLIONG]

Unity integration is considered done!

• Unity Native Plugin: https://github.com/taichi-dev/taichi-unity2
• SDK and examples: https://github.com/taichi-dev/Taichi-UnityExample