Evaluation of Marsden Fund shows Scientific Output Increased

Oct 2, 2015

New Zealand spends less money on research, relative to its size, than three-quarters of the countries in the OECD.

“The government is considering expanding public funding to narrow this gap, but very little has been known about the efficacy of existing funding mechanisms until now,” said Dr Adam Jaffe, Director of Motu Economic and Public Policy Research.

The Marsden Fund is the premiere funding mechanism for blue skies science research in New Zealand. In 2014, $56 million was awarded to 101 research projects chosen from among 1222 applications from researchers at universities, Crown Research Institutes and independent research organizations.

A study by researchers at Motu - a not-for-profit, non-partisan research institute – found that Marsden Funding does increase the scientific output of the funded researchers.

“What we found is that the public expenditure on the Marsden Fund is effective in increasing scientific outputs. A team that is given Marsden funding shows a 6-12 percent increase in their academic publications and a 13-30 percent increase in the papers that cite their work,” said Dr Jaffe.

Simulated Impact of Funding on Researcher Citations

“Publications and citations are, of course, only proxies for research output. However, we expect successful research to be highly cited,” said Dr Jaffe.

In an unexpected, but fascinating twist, the researchers also found no evidence that the selection process was able to meaningfully predict the likely success of different proposals.

The application process for the Marsden Fund has two stages. An initial one-page proposal is reviewed by a subset of the appropriate panel and given a preliminary score. 71-84 percent of the proposals are rejected. In the second stage, longer proposals are submitted and sent to external (typically international) anonymous referees for review. Applicants are given the chance to respond to referee comments before the panel scores and ranks the proposals.

“Interestingly, we didn’t find a link between a project’s future success and the rankings given to it by the second-round panel,” said Dr Jaffe. “This means there is no reason to expect diminishing returns if Marsden funding were increased. It also means the significant resources devoted to the second round evaluation could be reduced without degrading the quality of decision-making.”

This analysis is based on 1,263 Marsden proposals who reached the second stage of review between 2003 and 2008. Overall 41% of the second-round proposals were funded. Around 25% of the proposals were Fast Start (funding for early-career researchers) and slightly more than half of these were funded.

The average researcher on these teams made six proposals and received 1.2 grants between 2000 and 2012. Motu researchers also identified the approximately 1500 New Zealand based researchers named on these proposals and examined their annual publication and citation record between 1996-2012.

“We were very foturnate to have access to all the funded and unfunded proposals, including their evaluation scores. This meant we could control statistically for potential bias driven by the Fund’s efforts to fund projects that are expected to be successful,” said Dr Jaffe. The Marsden Fund and the Royal Society should be commended for making these data availabe, something very few research funding agencies do.

The working paper “The effect of public funding on research output: The New Zealand Marsden Fund”, was funded by the Motu Research and Education Foundation, Queensland University of Technology and the Ministry of Business Innovation and Employment.