Welcome to the HL2-DINO landing page. In some of our research at the Mechatronics in Medicine laboratory at Imperial College London, we've been exploring using AR headsets such as the Microsoft HoloLens 2 for tracking surgical instruments equipped with infrared reflective markers, which are commonly used in computer/robot-assisted surgery.

This project aims to enable the HL2 to track surgical tools with retroreflective markers

The HoloLens 2 contains a front facing time-of-flight depth-camera used for hand-tracking and spatial mapping. By using the Research Mode API, it is possible to access the underlying infrared response and depth map streams for this sensor. As part of our research, we have written software to use a combination of the Research Mode API and other libraries to enable the HoloLens 2 to detect of rigid tools equipped with flat retroreflective markers.

This is a set of multiple repositories which shares the image-processing system used in our publication: Semi-Automatic Infrared Calibration for Augmented Reality Systems in Surgery by Hisham Iqbal & Ferdinando Rodriguez y Baena.

DINO-DLL is a C++/WinRT project which interacts with Microsoft's Research Mode API and runs image-processing operations (enabled by OpenCV 4.5.3 and Eigen) on the raw sensor images to detect and estimate pose (position and orientation) for tools fitted with retro-reflective markers. The pose-estimation is done relative to the world-frame of the headset, and relies on the outputs of the self-locating algorithms which run by default on the HoloLens 2.

The project is set up to produce a .dll and .winmd which can be consumed in a Unity environment. When originally setting up the structure of the project, I found this GitHub project HoloLens2-ResearchMode-Unity by petergu684 to be an excellent reference for how to interact with the Research Mode API.

Image processing carried out onboard the HoloLens 2 via DINO-DLL

DINO-Unity is a sample Unity project, demonstrating how you can consume the information exposed by DINO-DLL in the context of a C# Unity project. If you 3D print and assemble the tools provided in DINO-IR-Tools, and then run the project on your HoloLens, you can get a hands-on experience of tool tracking. The repo provides further instructions for how you can track custom geometries and your own tools.

DINO-Unity user app experience recorded by Mixed Reality Capture

DINO-IR-Tools is a repo with 3D models and instructions for printing two example tools which are used in the Unity demos in DINO-Unity. It also provides some files which are intended to enable you to make your own 3D printed tools fitted with reflective markers for research and experimental setups.

Left: An exploded view of a tool with flat IR-reflective markers. Right: A tool being assembled

This work was originally developed during my PhD in the Mechatronics in Medicine Lab at Imperial College London. The original software was developed during a number of studies which were supported by INNOVATE UK (ref: 103950) and the EPSRC (ref: EP/R511547/1). Several people were instrumental in helping with different aspects of setting up the project and in the efforts to open-source the project. More detail on individual contributions is provided within each repo.

Thanks go out to:

Prof Ferdinando Rodriguez y Baena, Arjun Muhunthan, Andreas Keller, Dr Stefano Galvan, Joseph Ball, Dr Fabio Tatti