A new augmented reality system lets physiotherapy students see inside the human body by projecting different layers of muscles and bones over the top of a volunteer “patient.”
The technology, called the Augmented Studio, is designed to enhance the teaching of physiotherapy, in which students currently use their knowledge of anatomy to understand how muscles work beneath the skin of patients they can’t see into. But the Augmented Studio bridges the gap between that theory and practice.
‘Underneath our skin’
By using tracking sensors mounted on a scaffold it projects images of our muscles and skeleton directly onto a volunteer. The images automatically follow the shape and movement of the body, giving students in the studio space an interactive all-round view of how our bodies work. It can even allow them and their teachers to “draw” on the projected image to make information and action more explicit.
“What we are doing is overlaying virtual models of what we look like underneath our skin and synchronizing that with real human action,” says Thuong Hoang, a research fellow at the Microsoft Research Centre for Social Natural Users Interfaces at the University of Melbourne.
The Augmented Studio was built by Hoang, computer engineer Zaher Joukhadar, and doctoral student Martin Reinoso, who adapted Microsoft’s Kinect body sensing and tracking device as well as “RoomAlive” projection technology; both of which were originally designed for computer gaming.
Once a person steps into the projection space and forms a T-shape with their arms outstretched, the trackers lock on to them and the projected image conforms to their shape and movement.
At the moment the projected “overlay” doesn’t show how our muscles actually move when we contract and relax our muscles. Instead, it tracks the body and movement at the joints. But eventually Hoang wants to add in “animation” that can show the actual movement of muscles as the model moves.
Physiotherapy lecturer David Kelly says the students quickly embraced the technology during pilot sessions in 2016, which are continuing in 2017. He says the combination of live movement and interaction, in which students could actually move and feel the model’s limbs, helps them to grasp the relationship between their learned anatomy and how it works dynamically.
“For first year students it can be really hard to bring together anatomical knowledge with how the body actually works because it can be difficult to visualize. But when they see a real person who they can interact with, while also seeing the muscles and skeleton projected over the top, combined with the ability to draw and write on the body, it all becomes much easier for the students to learn about how the body moves,” says Kelly, from the university’s School of Health Sciences.
Tactile learning
The Augmented Studio also provides a more visual and intuitive way of learning that Kelly says will benefit those students who naturally learn more easily by direct visualization, rather than through reading and listening.
“There has always been a group of students that struggle because the limited ways in which we have to teach may not conform to how they learn best,” he says.
Developments in AR, which seeks to use technology to enhance what we can already see, hear, and feel in the real world, are far ahead of chasing GPS-tracked Pokémon. There are viewing devices such as glasses that can overlay what we see with three-dimensional graphics, video, and holograms, and we can generate projections like games that people manipulate by moving our hands.
The big advantage of the Augmented Studio over advances like 3D holograms is that the students can actually touch and move the body, making it a much more interactive experience. They also don’t have to wear headgear, which means it could potentially be used in bigger settings with larger numbers of students.
Augmented reality guides surgeons on the battlefield
“It has always been hard to capture the dynamic side of how our anatomy works, so the difference here is the high level of interaction you can achieve. The student can, for example, ask the model to kick and they can then look at variations from different angles at what is happening as someone kicks,” Kelly says.
The Augmented Studio is still in early-stage development and Kelly would love to see it migrate to using muscle animations. Hoang is also working to develop a system for the student interaction with the model to be automatically recorded onto their tablets so they can have a permanent record of what they were learning.
Another challenge is to find a way to make the studio more transportable and quicker to set up. At the moment the studio can work very effectively in a dedicated tutorial space where it could be permanently set up, but Kelly says a more portable set up would increase its flexibility for teaching.
Martial arts
Hoang is now working on extending the tracked projection technology to various health and fitness areas, and even in performance art. He says that using tracking sensors with projections it is possible to create guides that show people how to position their bodies for practicing fitness, sport, and dance.
Do cadavers make the best anatomy teachers?
Using virtual reality headsets he and doctoral student Martin Reinoso have already developed a prototype that allows a martial arts teacher to remotely instruct students on the right position to hold. By using body tracking and linked headsets student can match their movement to align with those of their teacher.
There is also scope to project information on our own body’s performance, such as heart rate and breathing, so it is visible either on our projected selves or on a nearby surface.
“The innovation we have created isn’t just limited to the fixed information that we have been projecting so far. If can be used to project dynamic information onto yourself or any surface around you,” Hoang says. “All of what I’m dreaming of is very possible.”
Source: University of Melbourne
