Apps to drive how soldiers use Army’s ‘mixed reality’ device

RESTON, Va. — In the not-too-distant future, soldiers at the lowest tactical level might carry a single device into combat that allows them to navigate, see in thermal or standard night vision, receive orders, fly a microdrone, and provide targeting coordinates for a strike mission.

As batches of the newest version of the Army’s $22 billion Integrated Visual Augmentation System — a combined augmented reality and night vision device — arrive at units, a vast team of military and civilian scientists as well as researchers will continue to tweak, refine and improve a piece of gear that could radically alter the soldiers’ capabilities.

The service has a lot to work with: The Army received 5,000 IVAS 1.0 devices over this past year. The service ordered 5,000 IVAS 1.1 devices in December 2022 and plans to begin fielding IVAS 1.1 by the fourth quarter of fiscal 2024, Program Executive Office Soldier told Army Times.

Those 1.0 and 1.1 versions are mostly slated for training units and schoolhouses to further develop how soldiers will learn to use and employ the device, officials said.

The 1.2 version is most likely to field for operational use and eventually combat. Twenty of the 1.2 versions arrived this past summer.

Another 80 such devices are planned for delivery in FY24, with 200 more devices headed to soldiers for testing in the first quarter of FY25, which is when the service plans to begin fielding IVAS 1.2, officials said.

PEO Soldier and Microsoft teams working on the IVAS program have made substantial changes to the device, some of which will continue to see improvements between the 1.0 and 1.1 versions as well as the 1.2 model.

The HoloLens-based iteration of IVAS emerged in March 2019, followed by ruggedized, military versions through mid-2022. As teams developed the device, it transitioned from a helmet/no-helmet goggle option with a chest-mounted controller and thick cabling that proved cumbersome for soldiers in various field tests. That development pushed the boundaries of technological territory, especially with combing digital technology and night vision. Current night vision technology remains mostly analog, relying on nearly 70 years of research and development.

Developers at PEO Soldier, along with Brig. Gen. Christopher Schneider, who runs the office, said the analog approach has reached its limits. That’s especially true for putting augmented reality, a digital technology, in the mix.

Challenges emerged early, including a staticky display and a low-light sensor that couldn’t provide acceptable night vision when compared to state-of-the-art analog systems that have advanced over decades of research. Other difficulties includes software and hardware reliability, eye strain, and nausea among a small subset of soldiers who used the device, and the form factor (or design of the device).

Over the past two years, the team has resolved all those problems, leading to recent approval by senior Army acquisition leaders to approve the second phase of 1.2 development, said Lt. Col. Denny Dresch, IVAS product manager for PEO Soldier.

To fix display distortion and software glitches, teams recoded and revamped software, built a new display, and found a better low-light sensor. They also added new coatings to the display to resolve humidity problems in wetter, hot-weather environments, Microsoft officials said.

Switching from a strap-on goggle that could be worn with or without a helmet to a more standard, helmet-mounted, flip-up visor version, much like existing night vision, added peripheral vision for the user and the ability to wear eye protection or eyeglasses while using the device, not previously an option with the goggle, Dresch said.

Changes to the device also reduced the light escaping from it, but developers also added a gasket-seal option and a blackout shield to put over the viewer so that no light escapes when using it at night.

Engineers moved the controller to a shoulder or rib cage mounting option, opening room on the chest. A slimmer, shorter cable connects the front heads-up display to the rear helmet-mounted computer, which now serves as a counterweight, much like those used by helicopter pilots wearing night vision.

Cable management and ditching the goggle strap came out of soldier touchpoints, events that involved troops using the gear and sharing their observations.

More recently, Dresch told Army Times, testing by 10th Mountain Division in August worked through more testing with both the low-light sensor and thermal vision, identifying targets at intervals between 50 meters and 500 meters.

For eye strain, nausea and improved visual quality, the team has worked a series of efforts almost since the inception of the program. Software fixes allow the system to improve “reprojection” so that the eye’s focal points are matched with the picture, reducing eye strain. The system also provides an artificial horizon, which reduces or eliminates nausea, much like how viewing the horizon aboard a ship can reduce seasickness. Improvements to color selection and brightness also reduced eye strain. A user-defined alignment tool now helps wearers adjust their visual alignment.

The digital technology used by IVAS is still in its nascent stages but already competes with both fielded night vision devices. Developers have included night vision and thermal view options in the device, and a fused mode combines both technologies.

The view pairs the Army-developed Family of Weapon Sights technology into the view. That allows a user to toggle between standard night vision, thermal and fused modes as well as select from traditional thermal view to a so-called outline thermal view. The outline view appears with nearly sketch-like lines of the object or person. It’s clear enough of a view to identify key features on a person, such as uniform, rank and face shape.

As recently as last year the team still had trouble creating training scenarios outdoors without using the interior walls of a building for reference points. But work over the past year has allowed users to place QR codes outdoors to achieve the same virtual shoot house or other scenarios as they would indoors.

In a demonstration with HoloLens technology, Microsoft personnel showed media at the company’s Reston, Virginia, office in September a scenario in which users created a terrain map of their objective using a combination of drone flight scans from a LiDAR-enabled drone for light detection and ranging; weather information from dedicated satellites; on-the-ground reconnaissance by the team; and information pulled from their own network as well as an offshore ship.

They first viewed a 2D map that transformed into a full-color, 3D model complete with layers of data such as the length of roads, trails’ locations, the number of buildings in the area, contour and elevation lines, and buildings’ interior blueprints.

Users were able to view the entire area and zoom into specific points for more detail. Visual detail was acute enough to view tire marks on dirt roads.

In the fictional scenario, the team identified hostile targets while on patrol and later conducted a strike.

Both PEO Soldier and Microsoft personnel pointed to evolving capabilities with the device through plug-ins and applications under development by engineers and soldiers. The Army has created Watchtower in recent years, a kind of app store for its Tactical Assault Kit. There are currently more than 100 applications available through the portal.

Schneider, the commander of PEO Soldier, told Army Times that it’s from those apps a lot of unanticipated, creative solutions for IVAS users will come.

How soldiers access many of the features will depend not on the device’s technology but more on how the Army decides to equip units with tactical cloud access and how planners expect to use the network.

“IVAS is not just what you’re wearing,” Schneider said. “A lot of the tools reside on IVAS already. A soldier can fly a soldier-borne sensor right now, take imagery of a target, send that to the tactical cloud package, which turns it into a 3D rendering.”

That capability allows dismounted soldiers to rehearse in near-real time actions on an objective moments before a mission, with a current view of what they’ll face at that location.

IVAS systems run as standalone devices, but capturing much of what a user sees, sharing information or downloading new apps in the field will come from the company-level computer system. The briefcase-sized device can be carried or mounted on a vehicle. It includes Microsoft’s Azure Stack Edge data transfer system, radio plug-ins and other slots for networking at a variety of levels.

Ultimately, power and bandwidth considerations will drive much of what’s available at what echelon for IVAS users.

Currently, only team leaders and above use Nett Warrior, a smartphone-based Tactical Assault Kit system for route planning and other unit data needs. But with IVAS, each member of the squad will use the new device.

That means a heavier power consumption and bandwidth burden for units. The device comes with three wearable, conformable batteries. The use strategy would have one battery on the device, a second charged battery in the ruck and a third battery charging for later use.

Schneider envisions a host of future options for the device’s use.

“The system is built so we can easily get input. Its easy to build an app to deliver the soldier something they want to do,” Schneider said.

A basic example might be fields of fire for a machine gunner preloaded onto the device for when that soldier reaches the fighting position. Developers could create an app that spits out a nine-line report for medical evacuations or an augmented reality option for medics to perform field surgery at the direction of a doctor at another location, he explained.

“It’s a night vision fighting device first, but it allows you to do all your other tasks more effectively, more efficiently,” he said.