全球智库信息集成服务系统

Broadly speaking, research organizations use three types of portfolio-level metrics

• Those are (1) aggregations of project-level data, derived by adding up data from individual projects; (2) narrative assessments; and (3) general (e.g., population-level) metrics, which might or might not depend on project-level data.

Nonmilitary organizations appear to be more focused than other organizations are on the measurement of research results beyond output measures

• Entities outside the U.S. Department of Defense (DoD) expended more effort than those inside DoD on measuring outputs and the ultimate consequences of portfolio investments — outcomes and impacts — than on the inputs and processes, which are perhaps more easily measured.

There is no one-size-fits-all approach to portfolio assessment

• Noteworthy innovative work in this area is taking place across research organizations.
• Evaluation at the portfolio level faces challenges similar to those at the project level, including research time lags, attribution and contribution issues, and data-collection burden. In addition, portfolio-level assessments need to contend with the challenge of portfolio heterogeneity.
• There are trade-offs associated with an organization's use of various assessment metrics, constructing an appropriately balanced mix of metrics needs to take into account organizational context, resources, and objectives for the evaluation of its research.

The effectiveness of research, like that of other activities, can be evaluated at different levels — the individual project, a group of projects or program, or a larger grouping that might include multiple programs (a portfolio). Focusing on options for research portfolio evaluation, RAND Corporation researchers found many metrics in use or recommended for federal agencies and private, research-supporting organizations and organized them in a taxonomy. This report presents the characteristics and utility of these metrics, organized by individual stages in a logic-model framework, mapping portfolio metrics to the upstream stages of inputs, processes, and outputs and the downstream stages of outcomes and impacts. At each stage, categories of metrics are composed of sets of metric types, each of which is, in turn, composed of individual metrics. In addition to developing this taxonomy, the authors appraised key attributes of portfolio evaluation metrics and described the trade-offs associated with their use. This structured, annotated compilation can help the Defense Health Agency and other entities that evaluate research portfolios to select, develop, or revise the metrics they use.

• Chapter One

  Introduction

• Chapter Two

  General Findings and Considerations

• Chapter Three

  Overview of Identified Metrics

• Chapter Four

  Conclusions for the Psychological Health Center of Excellence, the Defense and Veterans Brain Injury Center, and the Defense Health Agency

• Appendix A

  Variety in Evaluation Frameworks and Tools

• Appendix B

  Suggested Prioritization of Metrics

• Appendix C

  Additional Data on Metrics

• Appendix D

  Stakeholder Interview Topic Guide

• Appendix E

  Stakeholders Consulted for the Study

• Appendix F

  The Research Portfolio Management Data Dictionary of the Former Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury