A Framework for Evaluating the Beneficial Impacts of Publicly Funded Research

Many public entities are interested in empirical estimation of the overall impact, in terms of benefits to citizens, of publicly funded research. These include:

• increased incomes,
• better health,
• cleaner environment,
• enhancement of social and cultural values, and
• any other benefit that could be an objective of public policy.

The ultimate objective of this analysis is to develop reliable measures of such impacts in order to inform decisions about the level of public resources to devote to research, and to evaluate the relative effectiveness of different modes and mechanisms of research support.

Because different types of impacts are fundamentally non-commensurable, it is not possible to derive a single composite metric of all research impacts that would be useful for decision purposes. Further, for some important impacts there may be no meaningful quantitative measure, or there may only exist metrics that are illustrative or indicative of the impact in some approximate way. Any evaluation that excluded those impacts that cannot be directly quantified would be biased.

For these reasons, the approach of this paper is to identify a set of metrics and indicators that broadly covers the major categories of impacts. In each case, metrics are proposed that come as close as possible to direct quantification of the impact of interest, but where indirect or approximate indicators are the only ones available for potentially important impacts, they are included.

The report addresses a number of cross-cutting issues that arise in any attempt to evaluate the long-run impacts of research. These include:

• the need to distinguish between true objectives and intermediate outcomes,
• the need to distinguish between the impact of a particular investment and benefits that would have occurred anyway (the “treatment effect”),
• the long and uncertainty delay between research and its impacts,
• irreducible uncertainty about impacts, and
• the question of whether an attempt should be made to measure all impacts in monetary terms.

Although these problems cannot be truly solved, the report suggests approaches for dealing with them.