全球智库信息集成服务系统

Overall, the evidence is this. Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement. Considering the scope and costs of peer review internationally, this gap should be addressed.

The evidence that is available suggests the burden of peer review largely falls on applicants (rather than reviewers), but this typically receives less attention since it is less visible to funders. Additionally, application rates are increasing, so there is a need to reduce applicant burden, perhaps though reducing the complexity of the application process (though careful evaluation would be needed to ensure this is effective). In terms of effectiveness, one key challenge is determining how 'good' is 'good enough' in terms of reliability, discernment or bias. Clearly there is uncertainty in peer review decisions, but most funders do not use or even acknowledge this in their assessment processes. There is clear evidence that peer review can stifle innovation, and that it is not a good predictor of future (bibliometric) performance, which may be related to the broader pool of evidence that peer review ratings are inconsistent. There is more limited evidence of bias in peer review in relation to cognitive familiarity and due to cronyism.

Though often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009, RAND Europe conducted a review of the literature to evaluate these criticisms. This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes.

• Chapter One

  Introduction

• Chapter Two

  Methodology

• Chapter Three

  Grant peer review: the context

• Chapter Four

  Systems of grant peer review: international practice

• Chapter Five

  Evaluating grant peer review: what do we know?

• Chapter Six

  Conclusions

• Appendix A

  Case study template and interview protocol

• Appendix B

  Case studies