At I/ITSEC 2019, BISim demonstrated a new mixed reality AH-64 helicopter simulator, leveraging best-of-breed components, to showcase the power and training readiness of mixed reality.
“When we started looking into virtual reality we thought training could be more affordable and immersive using VR,” says Otakar Nieder, BISim’s Senior Director of Development. “It is more affordable than typical dome-based solutions and we found that the VR headsets have a big advantage of 360-degree views, but the problem is you can’t touch the virtual world. VR can be good for familiarization with the cockpit and training some basic procedures, but it’s not natural.”
Mixed reality emerged as the next logical step from virtual reality, says Tim Turcich, BISim’s Sr. AR/VR Engineer.
“Mixed reality is connecting what you are doing in the real world with the virtual world,” Tim explains. “Let’s say you were just doing virtual reality and lined up a physical chair with a virtual one. It would be a form of mixed reality. You could touch it, sit in it and be in virtual reality. But the visual doesn’t pass through the headset. Sight is one of the most powerful senses. That’s why a passthrough camera is so powerful.”
With a high-detail Apache cockpit from Bugeye Technologies, the Varjo XR-1 Developer Edition (the world’s only photorealistic mixed reality headset), a simulation host from Zedasoft and BISim’s VBS Blue IG as the high-fidelity virtual environment, I/ITSEC attendees could see the physical cockpit, cyclic and collective, console buttons, and see their own hands turning dials in the cockpit while also seamlessly viewing the virtual environment merged with the physical world.
“At this point today, Varjo is the only headset that has see-through capabilities that can be used for our use case because it offers high resolution and low latency,” Otakar says. “There is nothing else like it on the market.”
With the passthrough camera, pilots and other attendees at the booth tested out the quality of visuals by pulling out their notepads as they would in flight and started writing and reading while wearing the headset.
“With entry-level training you have pilots looking at the instruments more, eventually training extends into muscle memory,” Tim explains.
“The Bugeye cockpit offers 1-to-1 controls and multi-functional displays, so that pilots can get into the cockpit and feel as close as you can to the real Apache,” says Tim. The Bugeye hardware uses a control loader to provide linkage and accurate stick feedback that increases the immersion, he adds.
BISim has already successfully delivered a similar system to the U.S. Navy for its T-45 simulators using both mixed reality and virtual reality approaches. BISim’s T-45 solution, also on display at ITSEC with the VRgineers’ XTAL and Valve Index headsets, was used to retrofit the Navy’s T-45 cockpits (previously used with dome displays) to use Varjo headsets, VBS Blue IG visuals and FLEX-air, which serves as the simulation host and provides all systems modelling, avionics, flight models, animations, and interactions.
“High detailed 3D environments rendered by VBS Blue IG help extend the immersivity with 3D trees and real-world view distances, so pilots are training in more realistic environments,” added Otakar.
Tim notes that spatial understanding developed by tracking plays an important role in helping put together effective mixed reality solutions.
“I think we’re in an interesting time where tracking technologies are doing things for virtual simulation that enable more advanced mixed reality applications,” says Tim. “Lighthouse tracking has become much more cost effective and we haven’t even touched the tip of the iceberg for calibrating simulators.”
In addition to close attention to tracking and the high-fidelity VBS Blue IG virtual environment, Otakar notes that BISim’s approach to masking the real-world sets it apart from others. “Our dynamic 3D mask gives us more control over what we make and how users move in and out of the cockpit,” he explains. “A static mask would have the same cutout if you stepped out of the vehicle where if you stepped out of the simulated vehicle with the headset on you would see the exterior of the vehicle.”
“I feel like it’s history in the making,” Otakar adds. “After 3 years of developing this, customers can go and use this for training right now. This is a gamechanger.”