Just came on this in Aaron Sloman’s wonderful home page:

What should I write if asked to act as an academic referee, and the invitation requests me to assess the impact of the candidate’s work?

“I am not interested in impact, only quality of research, which does not always correlate with impact, since the latter is often subject to fashion and transient funding policies, etc. If you want a study of impact you would do better to consult a social scientist. Moreover high impact as measured by citations is often a consequence of making mistakes that many other people comment on.***

Many great research achievements could not possibly be assessed for their impact until many years later, in some cases long after the death of the researcher (e.g. Gregor Mendel).I’ll tell you what I think about the research quality: things like the depth, difficulty, and importance of the questions addressed, the originality, clarity, precision and explanatory power of the theories developed, and how well they fit known facts, as far as I can tell. If there are engineering products I may have some comments on their contribution to research. I shall not be able to evaluate their contribution to wealth or happiness.”

Thanks to Seb Schmoller for the link.

***Footnote: Really acute observation this. One of the most cited research papers in recent history is the infamous MMR paper published by the Lancet.