全球智库信息集成服务系统

RL can tackle complex planning problems but still has limitations, and there are still challenges to this approach

• Pure RL algorithms can be inefficient and prone to learning collapse.

• Proximal policy optimization is a recent step in the right direction for addressing the learning collapse issue: It has built-in constraints preventing the network parameters from changing too much in each iteration.
• ML agents are capable of learning cooperative strategies. In simulations, the strike aircraft synergized with jammer or decoy effects on a SAM.
• Trained algorithms should be able to deal with changes in mission parameters (number and locations of assets) fairly easily.
• Few real-world data exist on successful and unsuccessful missions. Compared with the volumes of data used to train contemporary ML systems, very few real missions have been flown against air defenses, and virtually all of them were successful.
• For analyses involving the use of large simulations in place of large datasets, the required computational burden will continue to be a significant challenge. The scaling of computational power and time required to train realistic sets of capabilities (dozens of platforms) against realistic threats (dozens of SAMs) remains unclear.
• Developing trust in AI algorithms will require more-exhaustive testing and fundamental advances in algorithm verifiability, and safety and boundary assurances.

U.S. air superiority, a cornerstone of U.S. deterrence efforts, is being challenged by competitors—most notably, China. The spread of machine learning (ML) is only enhancing that threat. One potential approach to combat this challenge is to more effectively use automation to enable new approaches to mission planning.

The authors of this report demonstrate a prototype of a proof-of-concept artificial intelligence (AI) system to help develop and evaluate new concepts of operations for the air domain. The prototype platform integrates open-source deep learning frameworks, contemporary algorithms, and the Advanced Framework for Simulation, Integration, and Modeling—a U.S. Department of Defense–standard combat simulation tool. The goal is to exploit AI systems' ability to learn through replay at scale, generalize from experience, and improve over repetitions to accelerate and enrich operational concept development.

In this report, the authors discuss collaborative behavior orchestrated by AI agents in highly simplified versions of suppression of enemy air defenses missions. The initial findings highlight both the potential of reinforcement learning (RL) to tackle complex, collaborative air mission planning problems, and some significant challenges facing this approach.

• Chapter One

  Introduction

• Chapter Two

  One-Dimensional Problem

• Chapter Three

  Two-Dimensional Problem

• Chapter Four

  Computational Infrastructure

• Chapter Five

  Conclusions

• Appendix A

  2-D Problem State Vector Normalization

• Appendix B

  Containerization and ML Infrastructure

• Appendix C

  Managing Agent-Simulation Interaction in the 2-D Problem

• Appendix D

  Overview of Learning Algorithms