HoloLens Project Brings Mixed Reality Into The World Of Surgery
- Technology Electronics
Yang Liú- 0
- 14 minutes read
The surgical field now has seen a major development. After Virtual and Augmented reality, now comes Mixed reality in the form of HoloLens project. This headset from Microsoft is thought to bring a revolutionary change in the fields of surgery across the globe.
Back in 2017, Microsoft’s HoloLens 2 mixed reality headset was used to operate a patient with a collarbone fracture. There were joint orthopedic surgeons, Professor Thomas Gregory from Paris and Dr. John Erickson from New Jersey. By performing shoulder arthroscopy the bone was repositioned. This process of arthroscopy involves placing a small camera in the joints to determine what is causing the patient’s shoulder pain.
Both the surgeons, Gregory and Erickson were connected to Gobbato’s headset via the “Microsoft Dynamics 365 Remote Assist” application, and the view was shared on the computer screen through “Microsoft Teams”. The patient and the holographic images generated by Gobbato through CT were visible. Among them, one showed the damaged clavicle of the patient, and another showing his healthy clavicle. Three surgeons from 3 different continents discussed how to approach the procedure, consulting each step and sharing each bone’s approaches.
“They were my partners helping me with the surgery,” says Gobbato. “We had a French perspective, we had an American perspective and we had a Latin American perspective. We had one quarter of the world inside the operating room.”
This procedure was part of a HoloLens project that focused on showing how doctors can be benefited from HoloLens 2, and how doctors from all over the world collaborate for work. Initiated by Gregory, who did his first surgery with the use of HoloLens 2. This edge included doctors and surgeons from 13 countries on 5 continents.
12 surgeons between November to January used HoloLens projects for performing a surgery that ranged from a knee procedure in the United Arab Emirates to a shoulder replacement in South Africa. While this was done others joined by Gregory took turns in observing the output. This project today culminates with a chain of roundtable discussions on the contribution of mixed reality in the health care sector.
For Gregory, this was a medium to convey the benefits of what he calls as “the smartphone for surgeons.” Many people throughout the globe access smartphones to receive information on various niches, but for surgeons that are not the case. The surgeons have little access to the patient’s data inside an operation setup.
The first HoloLens project was introduced in 2016. From that time Gregory saw it as a potential instrument for surgeons to use it for communication. All of a sudden he was able to use hand gestures and voice commands to call for information without touching the keyboard or mouse.
“It was what I was looking for for many years,” exclaims Gregory.
He is the chief of the “Department of Orthopedic and Trauma Surgery” at Avicenne Hospital in Paris. Along with that, he is also a professor of orthopedic surgery at the University Sorbonne – Paris – Nord.
“I realized that I was able to use the HoloLens as a computer or a smartphone to get any information I need when I need it, during surgery. That allows surgeons to be quicker, to be more efficient, and to improve performance.”
“I really think mixed reality will be used in the operating room for so many things in the future,” says Gobbato. “I am very excited to see what we will be able to do with it in the next few years.”
The HoloLens can be operated with hand gestures and also voice commands. This allows the surgeons to get a view of the three dimensional holographic images of the patient’s body that have been created from X ray or other scans. Those virtual images can be moved by the surgeons to get the view from different angles. Along with that, they can even access the patient’s data while the surgery is on and can even ask for other documents or contact other specialists during the surgery.
Dr. John Sledge On The HoloLens Project For Spinal Fusion Surgery
Dr. John Sledge, an orthopedic surgeon in Lafayette, Louisiana performed a 10 hours spinal fusion surgery with the HoloLens project. Sledge carried up X rays and scans from the lower back portion of the patient to discover the pieces of hardware that needed to be removed and find out what is the best position for his instrument to access them. He shortly loaded the images of his patient to explain the preparatory history to other surgeons.
Sledge confesses that without the HoloLens project he would have been limited to some of the things from the computer screen or the patient’s scan.
“It’s an enormously limited data set that’s available to me,” Sledge says. “With the HoloLens, I can pull up the images I want and make them bigger as needed. I have got all the images I need literally right in front of me, in whatever size and clarity that I need them to be.”
Along with that the HoloLens project is also useful if the surgeon is planning the surgery or even for training, says Sledge. For example, there can be instances where with a holographic illustration of a patient’s shoulder the surgeon can determine their next step.
Sledge also adds that despite printing a 3D image of the bone, he can use the Hologram to practice the operation on this. For training, three dimensional interactive models can be made of any surgical procedure.
“Medicine, particularly surgery, is still an apprenticeship. You watch a person operate 100 times before you are allowed to,” Sledge added. “But now you can have residents run through 100 operations on the HoloLens, complete with rare complications and their solutions. We can do worst case scenario training. With the HoloLens, we can make the problems occur and doctors in training can solve it yourself.”
Dr. Stefan Greiner’s Experience With The HoloLens Project
Orthopedic surgeon Dr. Stefan Greiner had never used the HoloLens project before. But he had agreed to participate with Gregory in the project as he is his good friend. Initially, Greiner was doubtful about the idea. He thought this device to be like any other device that will be used in rare cases.
But once he learned the use of it he started to see more possibilities of the same. He uses the device to see images while performing any orthopedic surgery to avoid looking up and down where the images are shown on the computer screen.
“I think it’s going to improve the precision of surgery and the safety of surgery because you don’t have to look somewhere else,” Greiner added.
Dr. Stefan Grenier ruins the “Department of Shoulder and Elbow” at Sporthapaedicum Regensburg which is one of the largest sports medicine facilities in Germany.
This device can also be used to elaborate and prepare patients for surgery by making them look at the holograms or animations.
“It is so much clearer if you can use a hologram in action, rather than just seeing a small picture,” he says. “This would be of great use for patients to understand what kind of surgery they are going to have.”
Dr. Ashish Babhulkar After Using HoloLens 2 By Microsoft
Another instance just like Dr. Stefan Greiner is Dr. Ashish Babhulkar. He also had never used HoloLens before he signed up for Dr. Gregory’s project. Dr. Ashish Babhulkar is the head of the Department of Shoulder and Sports Injuries at Deenanath Mangeshkar Hospital and Research Center in Pune, India. He did two surgeries of reverse shoulder replacement with Dr. Gregory and an orthopedic surgeon from the U.K.
“Imagine that tomorrow in Kenya, someone’s got a HoloLens and he wants to start doing the surgery. He doesn't have to come to India to learn it,” Babhulkar says. “We can walk him through the process without him making errors and help build his confidence. It’s going to revolutionize treatment in some remote areas.”
He further says that his experience with HoloLens has convinced him to believe that this will go long in the world of medicine.
Dr. Babhulkar says, “The device is so powerful that as we progress on it, every year we are going to innovate and add novel applications to it. Hopefully, we will be using it for all our cases eventually.”
About The Device
The HoloLens itself and its forerunner, the “Kinect line of motion” – sensing devices, both were created by Microsoft Technical Fellow Alex Kipman. He is an engineer who knows the need for devices that will merge the physical world with the digital sphere in a three dimensional way.
The very first HoloLens was launched in 2016, and then came HoloLens 2 in November 2019. This new device is well equipped with “sophisticated environmental mapping hardware” and an “eye tracking camera” that allows it to know the space that is being used and also the focal point of the user. An application known as Microsoft Dynamics 365 Remote Assist was designed that allowed HoloLens to connect with Microsoft Team so that users can connect to people all over the globe.
A decade ago it would have been impossible to think about surgeries using HoloLens and the use of mixed reality technology. But today in 2021 HoloLens is being used in every field of medicine. It is used from “virtual hospital rounds” for medical students to treating Covid 19 patients.
Elena Bonfiglioli, Microsoft’s managing director and regional lead for “Healthcare and Life Sciences” in Europe and the Middle East says that the use of HoloLens in surgeries is new and has been propelled by Gregory and other pioneering physicians all over the globe.
“The future of medicine is empowered by holographic computing because holographic computing gives the new dimension of visuals,” Elena added. “We have had the dimension of collaboration for a long time. We have had the dimension of data, powered by artificial intelligence. But we didn’t have something that is very typical of humans, which is visual enhancement. That’s why it’s so powerful. It’s a technology that in itself is a change agent, empowering those who can see the future.”
The First Ever User Of HoloLens: Dr. Thomas Gregory
Gregory’s department has been using mixed reality technology for almost a year now and has used that for almost all his surgeries. He is even called by his younger colleagues for assistance regarding the HoloLens projects.
Microsoft partnered with Gregory in January 2020 to launch their Institut Moveo, which is a center for research and teaching that is completely dedicated to artificial intelligence in surgery. This center is located at the “University Sorbonne – Paris – Nord. This accommodates scientists, surgeons, and also mathematicians to provide training on HoloLens surgery.
The future plans of the HoloLens project include a collection of HoloLens surgery data and building an algorithm based on that. Gregory views HoloLens as an instrument that can be used at a much lower cost than a “surgical robot” used for surgery and can make AI in surgery accessible for all doctors.
Conclusion
“With HoloLens project, we’ve got a tool to democratize this digital revolution in all operating theaters around the world,” concludes Gregory. “The impact of this is massive. I think in the future, surgeons will not be able to perform a surgery without HoloLens.”
