“The Virtual Sixth Sense” - Transforming Soldier Lethality through Artificial Intelligence & Augmented Reality

“The Virtual Sixth Sense” - Transforming Soldier Lethality through Artificial Intelligence & Augmented Reality

Scott Padgett

As increasingly intelligent computing technologies mature, the U.S. Army will have the potential to harness and diffuse capabilities down to individual Soldiers that will change how small units operate. Future tactical battlefields will consist of "automated systems that move too fast, and the factors involved are too complex for real human comprehension."[i]

This article introduces advanced artificial intelligence concepts as force multipliers to transform lethality of the individual warfighter and small close-combat units.  Specifically, we present a theory called the "Virtual Sixth Sense" (VSS) whereby intelligent machines extend the human warfighter's senses and knowledge and fuse these capabilities with Augmented Reality (A.R.) to achieve and maintain tactical overmatch.  VSS intends to ease the cognitive burden on the tactical warfighter.  We demonstrate its potential through a notional combat scenario, examine some of the specific technologies and infrastructure required to support these capabilities, and review their potential vulnerabilities.

Amazon Web Services (AWS), Google, Microsoft, Apple, IBM, and many other companies are investing billions of dollars developing their commercial-off-the-shelf (COTS) artificial intelligence (A.I.) technologies. Many of these A.I. technologies have utility on the battlefield in two broad categories. First, machines and intelligent algorithms can extend a warfighter's knowledge and human senses on the battlefield through Enhanced Sensory Perception (ESP). Second, with the right data and user interface, combat systems can pair A.I. with Augmented Reality (A.R.) technologies to improve Soldier situational awareness dramatically. Collectively, the marriage of these capabilities can transform and enhance a unit's knowledge and lethality by increasing the speed and accuracy of decisions and engagements.

Most of these commercial companies offer A.I. software developer workbenches whereby software developers, data scientists, and end users (warfighters) can access large portfolios of A.I. applications, “micro-services” (also known as “application protocol interfaces” (APIs)) and kits to invent, configure, integrate and build new mission-critical services like VSS.[ii]  These corporations are also making substantial investments for offering these A.I. services on public, private, and hybrid cloud infrastructure services that can extend to the warfighter’s tactical edge.  These companies are also working with government agencies to get their A.I. services accredited to operate across the DoD’s secure network enterprise.

Software factories scattered across the U.S. Army and the Department of Defense (DoD) can consume and integrate these A.I. micromachines to invent new capabilities to support the warfighter.  The emergence of software “containers” allows developers to package these collections of hosted A.I. micro-services into virtual weapons packages.[iii]  As new A.I. services like VSS go into full production, tactical warfighters will discover, access, and download these managed services from their unit’s “Virtual Arms Room” (VAR).[iv]

To visualize VSS’s potential, we can imagine an infantry squad as they execute notional combat operations against a near-peer adversary.

Begin Scenario

The squad arrives at a forward staging area and is set to advance soon as part of an envelopment to seize critical enemy infrastructure behind the front lines. Each squad member prepares their kit for combat, including tests of their body-worn computers and sensor arrays. The system verifies a secure connection to the secure tactical network, and each Soldier downloads the latest VSS AI Services Package before onward movement. The service package includes navigational and map data, updates to mission support tools, and the newest mission overlays.

A day later, the squad finds themselves conducting a hasty battle handover with an elite SOF unit that seized a ballistic missile site. While the weapons depot and launch sites are clear, the squad is tasked to expand the security perimeter by clearing nearby villages and establish a defense to prevent an enemy counterattack. The clearance moves quickly, but during entry to one building, the squad leader encounters a family huddled in a corner. His HUD highlights the face of a middle-aged man and indicates a 97% probability facial match of a High-Value Target (HVT) that eluded capture during the Special Operations raid the previous night. The squad leader calls the man forward and asks him a question in English through his whisper mike. The A.I. Language Translation Service[v] in VSS translates the question and projects the translation through a small on-body speaker.  The two begin conversing without a human interpreter. In the background, the possible HVT’s voice pattern is analyzed through A.I. Audio Background Classifiers[vi] and A.I. Speech Recognition[vii] tools.  The squad leader's “Intelligent Virtual Assistant” (IVA) named “Colossus” informs him that the combined facial recognition and voice match data indicate conclusively this man is a nuclear scientist working for the enemy's Inter-Continental Ballistic Missile (ICBM) program.

The squad leader calls a fire team into the building to take custody of the HVT and begin searching the building more thoroughly. He leaves only two of his Soldiers on guard in the alleyway enabled by a network of small air and ground drones (unmanned aircraft systems (UASs) and/or unmanned ground systems (UGSs) and 360-degree camera arrays linked to the tactical cloud’s A.I. Visual Recognition Services[viii].  Aided by Colossus, the buddy team is alerted to any movement and its direction within 300 meters and able to call live video into their HUD on demand. A.I. Rapid Target Acquisition (RTA) services highlight potential targets in the fire team’s HUD displays by putting little red digital boxes around emerging threats, and little green boxes around friendlies. The search team inside the building quickly uncovers stacks of papers and technical schematics and what looks like some type of trigger device.  The youngest Soldier’s HUD alerts him to the top sheet on an otherwise innocuous binder, translating the content into English using an A.I. Text Analytics Translation Service[ix], revealing it as a nuclear program technical pamphlet that has intelligence value.

Concurrently, the team leader, examines the trigger device.  Colossus notifies him through the A.I. Visual Recognition - Object Identifier API the device is the firing mechanism for a medium-range mobile missile and requires immediate disarming.  Observing the device through his HUD, the device's "image" prepares an Augmented Reality (A.R.) experience in the team leader’ HUD.  Engaging his special A.R. lens, he immerses himself into a mixed reality environment and performs a series of complicated disarming procedures on the device.  End of Scenario.

In this notional scenario, we demonstrated how a suite of integrated A.I. micro-services & A.R., securely accessible on a tactical network could provide an average infantry squad capability that even our most elite formations lack today.  All of the technologies described in this notional scenario are commercially available today.  This scenario highlights significant operational and intelligence benefits of A.I. for the basic infantry squad and individual warfighter.  We will now discuss several important technology concepts that are critical to support this concept.

Modes of Operations – Connected Compute Services

As our warfighters gain more reliance on machines to conduct combat, they will require a “basic load” of compute services to maintain tactical overmatch in disconnected or degraded network environments.  However, we prefer our soldiers to operate in "fully-connected-networked modes of operation" whereby body-worn computers are fully uplinked through next-generation networks (like 5G or 6G) to the tactical cloud.  By shifting a significant portion of the required computational services to the tactical cloud, we can reduce form factor and the size, weight, and compute power that a soldier must physically wear and carry into battle.  This is accomplished by leveraging a new class of next generational networks and modular/mobile tactical data centers that provide enterprise cloud services to the tactical edge.

AIOps – The Next Generation of Software Development Platforms & Methodologies

In the future, the tactical warfighter will be supported by a myriad of AI-infused smart applications like VSS.  As adversaries bring their own competing capabilities, the demand to rapidly evolve intelligent services will accelerate. The emergence of A.I. software development platforms, inclusive of a growing portfolio of A.I. micro-services serves as the core foundation by which the Army will rapidly invent and produce new innovative capabilities in support of the tactical warfighter.  The modularity of these A.I. micro-services enables developers and data scientists to integrate complimenting A.I. services together to invent new capabilities like VSS.  Additionally, we are seeing a fundamental shift on how unique and innovative A.I. services are being developed by including warfighters (subject matter experts) and testers earlier in the A.I. software design/development process.  Artificial Intelligence Operations (AIOps) and will serve as a preferred software development methodology as the need for quicker and better A.I. services are needed on the tactical battlefield.

Meet “Colossus” – Our Newest Virtual Squad Member

In our notional scenario, we introduced Colossus, the intelligent virtual assistant (IVA).  (Think Alexa, Cortana, or Siri).  The name of our IVA is immaterial, but its function is critical to our warfighters operating on the tactical edge.  This bot exists in our Soldiers’ wearable computers and on the tactical cloud, emulating human interaction to help our warfighters perform complex and accelerated tasks.  Its most important function is accelerating knowledge discovery through machine automation to keep our warfighters inside the enemy’s OODA Loop.  Another essential feature of the IVA is to assist the human in reducing the amount of data overload being thrown at the Soldier, and prioritizing decision making.  IVAs are increasing because of significant advancements in deep neural networks and machine learning.  The intelligence behind this technology provides the capacity for knowledge, reasoning, and understanding rapidly changing combat environments.  There will be a continued role for both, blending artificial with real human intelligence and finding the best balance to maintaining tactical overmatch.

Augmented Reality - Overlay Virtual Knowledge onto the Warfighter’s Reality

A.R. is being adopted by the commercial gaming and marketing industries to transform a user’s experience.  A retail marketing example of A.R. is the Jack Daniels Whiskey AR experience.  A customer views any Jack Daniels bottle their smartphone, and the label serves as the "A.R. Object Trigger" that automatically activates the A.R. experience in the application.[x]

Another concept mentioned in our notional scenario is "Augmented Reality" (A.R.). Our hypothetical team leader used A.R. through a special lens. A.R. can help our Soldiers navigate, visually identify HVTs, and guide them through technical tasks that would otherwise be beyond their skill level. A.R. involves digital information overlaid onto the real world through their HUD. The promise of A.R. in support of the tactical warfighter are endless. A.R. can be adapted to support a myriad of operational and training functions to include synthetic training environments (STE), mission rehearsals, targeting, holographic target displays, telemedicine, among others.  The Army can create a library of “digital object classifiers” that serve as the "A.R. object triggers" to initiate A.R. experiences for the warfighter. 

Some Potential Vulnerabilities

The introduction of a new generation of weapons and systems enabled by A.I. and A.R. brings on a new generation of vulnerabilities.  Some of these vulnerabilities apply to the applications, cloud, network, and bandwidth layers that make up the system’s support architecture. 

Future tactical warfighters will wear body-worn systems (computer, network, visors, lenses, cameras, sensors, speakers, power and microphones) that can talk to each other, collect data from their physical surroundings for automation, monitoring and analytical purposes.  The key to protecting these tactical warfighter systems is building redundant protections and security protocols into every aspect of the system’s architecture and start as early as possible in the development phase.

Our enemies will be searching and probing for vulnerabilities in these systems by exploiting security weaknesses across many different layers of these interconnected and complex systems to include the hardware; web, mobile, network, radio frequency, and cloud components.  As an example, we should explore emerging security technologies like Blockchain to protect our systems between the sensors and managed A.I. cloud services running on the tactical network.  “Blockchain technology, for the first time in history, enables various non-trusting members to interact with each other through a peer-to-peer network in a secure and verifiable manner. These disruptive characteristics attract much curiosity throughout various industries. Transactions which were formerly only possible through intermediary oversight can now be conducted in a decentralized consensus finding network.”[xi]    

Another major vulnerability is the fifth-generation infrastructure network known as “5G”.  Strategic competitors such as China are investing considerable effort in becoming world leaders in telecommunications technologies like 5G networks. The U.S. must prioritize trusted development of secure communications platforms and technologies to make these emerging digital capabilities reliable on the battlefield.  This dovetails with another vulnerability called “radio frequency (R.F.) spectrum scarcity."  There is a finite amount of Radio Frequency (R.F.) bandwidth on the battlefield, and it continues to get more congested as more sensors, devices, autonomous weapons and A.I. machines are using it to communicate and exchange data in support of the warfighter.  The Army should examine using A.I. to perform automated spectrum management services versus today’s manual spectrum planning to maximize R.F. spectrum availability for the warfighter.              

Conclusion

This article introduced a series of innovative A.I. concepts as force multipliers to transform lethality of the individual warfighter and small close-combat units.  We presented our vision for the “Virtual Sixth Sense” that fuses a myriad of A.I. micro-services with A.R.  First, machines and intelligent algorithms that extend the warfighter’s human senses through Enhanced Sensory Perception (ESP). Second, with the right data and user interface, combat systems that pair A.I. with A.R. to improve Soldier situational awareness dramatically.  Collectively, the marriage of these capabilities can transform and enhance a warfighter’s lethality by increasing the speed and accuracy of decisions and engagements. 

The viewpoints, comments, and opinions provided in this article are strictly those of the authors.  None of the content contained in this document is in any way an endorsement or representation of the U.S. Federal Government, any Government agency, former employer, or any commercial company, product or service(s).

End Notes