Mixed Reality Flight Training

Virtual Reality-based training devices enable pilot trainees to practice more frequently due to their affordability and portability. As a supplement to traditional fixed training devices like dome-based or full flight simulators, VR-based training provides increased repetitions and sets which helps compress the learning timeframe and allows pilots to learn at their own pace. This optimizes the use and scheduling of limited full mission simulator time.

Mixed reality-based training is a step up in fidelity from VR. MR systems immerse the trainee in a synthetically generated virtual world which is seamlessly integrated with a physical aircraft cockpit. Trainees view the real world through high resolution pass-through cameras on the Varjo headset which enables them to interact with the “knobs and dials” of the facsimile cockpit, with its simulated and emulated systems. This allows for the development of muscle memory related to critical flight skills.

This highly effective blend of the physical world (e.g., pilot’s kneeboard and the tactile feel of the flight stick) seamlessly blended with the very high fidelity virtual world of the latest VBS products means training providers can accurately replicate the pilots operating environment in a low cost device. This truly transforms training by creating transferable skills obtained in a repeatable, reliable environment that replicates the demands of the aircraft.

“At Bohemia Interactive Simulations, we excel by working with industry leaders who are developing cutting-edge technology,” says Otakar Nieder, BISim’s Senior Director of Development. “Varjo currently provides the only headset able to support human-eye resolution combined with the video pass-through cameras and associated software features needed to support mixed reality training.”

Using ultra high-definition stereo pass-through technology, the Varjo XR-1 headset captures the real world and provides a variety of methods to composite the real and virtual world. The solution is enabled by BISim^TM developers who define an area or “mask” inside the VBS Blue IG that matches the shape of real-world objects like an aircraft cockpit. This “mask” creates a portal from the real world into the virtual world so both are seamlessly merged. Pilots wearing the Varjo headset can then see their hands and interact with the physical controls inside the real cockpit while simultaneously immersed in the virtual world scene created by VBS Blue IG.

“We are using several bespoke algorithms to combine the virtual and the real world and I haven’t seen anything like that anywhere else,” Otakar explains. “Our 3D dynamic mask process gives us more control over what we mask and how it moves within the cockpit. This also means that if you stepped out of the vehicle, you can view the exterior of the vehicle and still be in the virtual world.”

BISim created a concept demonstrator Mixed Reality (MR) device in December 2019 for the I/ITSEC conference, replicating an AH-64 trainer. The trainer hardware, provided by Bugeye Technologies, included individual cockpits for both pilot and co-pilot, with control loaded cyclic, collective and yaw pedals along with functioning MFD’s displaying unclassified data. Zedasoft provided the AH-64-D simulated OFP avionics and integration with Bihrle Applied Research flight dynamics simulations to offer a complete reconfigurable simulation solution. The pilot and co-pilot were wearing the Varjo XR-1 Mixed Reality (MR) head mounted displays (HMDs) displaying an immersive virtual environment using BISim’s VBS Blue IG. As the pilot/co-pilot flew the aircraft, the virtual environment displayed in the HMDs. As they looked down or anywhere within the cockpit, they saw their controls and instruments along with their hands, feet and anything else present in the cockpit. They viewed and interacted with their controls and instruments as needed to fly and navigate. This real content was seamlessly blended into the high detail synthetic environment created by VBS4 with the result being that it was very difficult to tell which part of the scene was real and which part was virtual.

“The BISIM developed Apache MR concept demonstrator shown at I/ITSEC 2019 was one of the most significant innovations in the industry in the past decade,” said John Burwell, Global Lead Simulation and Training at Varjo. “With Bugeye’s cockpits and Zedasoft’s simulation Apache OFP combined with VBS Blue IG in our mixed reality HMD XR-1, provided a world-class high-fidelity VR experience. This solution promotes mixed reality training where a user needs to see and interact with a physical cockpit or crew stations while immersed in a synthetic environment. BISim has already developed and delivered several very compelling examples of mixed reality for pilot training that offer a more cost-effective and portable alternative to more expensive traditional fixed flight simulators.”

For more information, visit varjo.com, bisimulations.com, www.zedasoft.com, www.bugeyetech.com, bihrle.com.