Project plan

Project Name: Magic Mover

Abstract: Magic Mover is an AR application meant to deliver a novel form of motor, balance and/or cognitive rehabilitation either in the hospital or at home. It exploits the AR technology capabilities to present the user a set of exercises in a gamified form, which could results to be more engaging and interactive with respect to the traditional rehab activities.

Background of the Project The ageing of the population and more generally speaking the growing demand for healthcare solutions that could be either more sustainable and more effective has already engaged the use of advanced technology solutions. Among these, the use of interactions with virtual objects in rehabilitation has been already fostered in many rehab fields. The recent availability of new relatively affordable AR products opens the way to their use in this field. In particular, Hololens and Magic Leap Ones may represent an interesting starting point to further foster this kind of applications, since they share the following features: -) do not separate subjects from their real world -) instead they could embed in the subject’s reality items that could be used either to promote rehab activities related to motion or (in the future) may reinforce the interaction of the subjects with their environment (i.e. cognitive impairment) -) they are portable, without the need of being tethered to a PC, thus leaving the subjects free to move around -) they do not require any environmental change

Given the novelty of these technologies, there is still very little scientific literature on their implementation in rehabilitation, hence highlighting the importance of further studies in the field, which could prove the effectiveness and highlight the limitations related to this use of AR, in particular HMD based.

Aim of the project The project is aimed at promoting specific motor activities related to the interaction with virtual object purposefully perceived by the subject, wearing AR HMD, as placed onto real assets in the room. Such activities are presented in a game like form, thus increasing the engagement, and may be modulated according the specific impairment and capabilities of the subject, requiring subjects to move their arm only up to their whole body to reach for virtual objects placed far away.

Post stroke patients are one of the potential target population, although given the nature of the system, many other types of diseases may benefit from this novel “embedded” approach to motor and cognitive rehabilitation. Another potential population is represented by post Traumatic Brain Injuries (TBI) subjects, who could benefit from an interactive set of activities tailored on their conditions.

Some quantitative information, related to the reaction times, the movements and the center of mass trajectory may be saved and transferred to a computer or a cloud based database, to keep track of the rehabilitation process remotely.

Mathods Subjects would wear an AR HMD (i.e. Hololens), and follow simple instructions appearing on their screen, and the interaction is completely controller-free (so they do not need to handle any weight). They would select the next activity to perform according the level they reached in previous sessions or pre-defined by clinician/therapist. Selection is made by placing the desired item at the center of the view area and then issuing an event according the particular equipment (i.e. pinching in Hololens). Once started, the HMD will place a virtual objects (VO) overlapping the real room and anchoring it in a specific predefined position. the activity will require the subject to interact with VO according basically three different complexity levels: Simply reach for and “touch” the object Simulate a grasp of the VO that must be moved to another place, indicated by a mat Simulate a series of grasps and moves according to a specific sequence (i.e. Hanoi Tower, etc.)

The task is repeated for a predetermined number of times and then, if the subject has accomplished all the tasks properly (i.e. below a certain amount of time), the main menu is presented, allowing the subject to select the previous level or an higher complexity one (this can be customized by the clinician/therapist).

If the subject do not perform any meaningful action within 10 second (figure can be parametrized), the object start emitting a spatialized sound cue, helping its localization. If the subject do not perform the next desired action (“touch”, “grasp”, “release”), after 30 seconds (figure can be parametrized), automatically a tutorial for that level appears, trying to help the subject to complete the task. After a certain amount of time, a timeout sound is played and the main menu is presented to the subject.

Limitation and future development The kind of exercises proposed to the subjects have some limitation at the moment. The first main limitation is related to the equipment used (hololens in the current release), which could not track the position of the hands, hence requiring the subject to move the head to align a virtual cursor in the center of the screen with the particular item (button on the main screen, VO, etc.). This represent a major limitation, in requiring the subject to operate non intuitive actions to touch and move objects.

The current tracking is limited also in terms of perceiving the positions of the subject’s body in terms of trunk and lower limbs, which instead are of key importance to be considered when dealing with rehabilitation.

The second limitation is related to the maximum distance that can be detected by the HMD, hence allowing to place objects (and anchoring them) far away from the subject’s initial position.

A third limitation may be represented by the narrow field of view offered by the HMD, which forces the subject to wide rotation of the head to explore the room

Finally, a fourth limitation may be represented by battery-powered nature of the HMD, which in order to be portable may have a limited number of continuous working hours before requiring to be recharged

Despite these limitation, there are many possible future interesting developments. The first would be to increase the complexity in the behaviour of the VO, or even their motion (now stationary), allowing more challenging types of motor activities. More complex sequence of interactions with Vo may also make possible to realize AR-based games like virtual treasure hunts.

The multimedia capabilities in recording and producing visual and acoustic signals can be also exploited, i.e. tracking voice, exposing the subject to audio/visual distractors, etc.

It may be also very interesting to “virtualize” real objects, thus allowing the subject to have also a physical interaction with them, along the virtual one. This could be of particular importance in trying to improve muscular activity (i.e. handling objects with real mass instead of moving the hand alone), bimanual activities, etc. Finally, the possible capability of detect the environment and tag (i.e. attributing semantic values) specific objects or pieces of furniture, may enable a totally novel class of ecological rehabilitation activities that may be performed at home by cognitive impaired subjects.

Future extensions of the projects may also be related to the pediatric field and the elderly.

Problems to address