G2TT
来源类型Report
规范类型报告
DOIhttps://doi.org/10.7249/RR-A464-1
来源IDRR-A464-1
Evaluating the Effectiveness of Artificial Intelligence Systems in Intelligence Analysis
Daniel Ish; Jared Ettinger; Christopher Ferris
发表日期2021-08-26
出版年2021
语种英语
结论

Using metrics not matched to actual priorities obscures system performance and impedes informed choice of the optimal system

  • Metric choice should take place before the system is built and be guided by attempts to estimate the real impact of system deployment.

Effectiveness, and therefore the metrics that measure it, can depend not just on system properties but also on how the system is used

  • A key consideration for decisionmakers is the amount of resources devoted to the mission outside those devoted to building the system.
摘要

The U.S. military and intelligence community have shown interest in developing and deploying artificial intelligence (AI) systems to support intelligence analysis, both as an opportunity to leverage new technology and as a solution for an ever-proliferating data glut. However, deploying AI systems in a national security context requires the ability to measure how well those systems will perform in the context of their mission.

,

To address this issue, the authors begin by introducing a taxonomy of the roles that AI systems can play in supporting intelligence—namely, automated analysis, collection support, evaluation support, and information prioritization—and provide qualitative analyses of the drivers of the impact of system performance for each of these categories.

,

The authors then single out information prioritization systems, which direct intelligence analysts' attention to useful information and allow them to pass over information that is not useful to them, for quantitative analysis. Developing a simple mathematical model that captures the consequences of errors on the part of such systems, the authors show that their efficacy depends not just on the properties of the system but also on how the system is used. Through this exercise, the authors show how both the calculated impact of an AI system and the metrics used to predict it can be used to characterize the system's performance in a way that can help decisionmakers understand its actual value to the intelligence mission.

目录
  • Chapter One

    Introduction

  • Chapter Two

    Tracing Effectiveness from Mission to System

  • Chapter Three

    Measuring Performance and Effectiveness

  • Chapter Four

    Conclusions

  • Appendix A

    Derivations and Technical Details

主题Artificial Intelligence ; Intelligence Analysis ; Military Intelligence
URLhttps://www.rand.org/pubs/research_reports/RRA464-1.html
来源智库RAND Corporation (United States)
引用统计
资源类型智库出版物
条目标识符http://119.78.100.153/handle/2XGU8XDN/524539
推荐引用方式
GB/T 7714
Daniel Ish,Jared Ettinger,Christopher Ferris. Evaluating the Effectiveness of Artificial Intelligence Systems in Intelligence Analysis. 2021.
条目包含的文件
文件名称/大小 资源类型 版本类型 开放类型 使用许可
RAND_RRA464-1.pdf(1066KB)智库出版物 限制开放CC BY-NC-SA浏览
x1629979821612.jpg.p(3KB)智库出版物 限制开放CC BY-NC-SA浏览
个性服务
推荐该条目
保存到收藏夹
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Daniel Ish]的文章
[Jared Ettinger]的文章
[Christopher Ferris]的文章
百度学术
百度学术中相似的文章
[Daniel Ish]的文章
[Jared Ettinger]的文章
[Christopher Ferris]的文章
必应学术
必应学术中相似的文章
[Daniel Ish]的文章
[Jared Ettinger]的文章
[Christopher Ferris]的文章
相关权益政策
暂无数据
收藏/分享
文件名: RAND_RRA464-1.pdf
格式: Adobe PDF
此文件暂不支持浏览
文件名: x1629979821612.jpg.pagespeed.ic.wPHSsrxgXV.jpg
格式: JPEG

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。