Section 11: How can you evaluate societal impact?

How can you evaluate societal impact?

Understanding how best to measure and evaluate the societal impact of research is clearly important to researchers we interviewed and surveyed. In particular, the more senior researchers we spoke to discussed how they are trying to incorporate societal impact into evaluation and assessment frameworks for academics earlier in their careers.

However, what’s also clear from both our survey results and interviews is that there is relatively little consistency between how researchers try to measure and evaluate impact.

Societal impact is, without a doubt, complex to understand and therefore to track. We believe that at least some respondents to our survey may not have fully understood the term societal impact, and that researchers may include academic impact as well when referring to the impact they sought to achieve or what they monitor to measure it.

It’s therefore perhaps not surprising to see that the researchers in our survey predominantly use academic impact measures to track societal impact. Nearly three quarters of respondents said they track citations, and half referred to downloads of their research papers as measures of societal impact.

In contrast, the researchers we interviewed spoke more about qualitative measures, with some stating explicitly that they felt citations were not a measure of impact. There was far more focus on keeping track of the number of collaborations they had and the qualitative stories they could tell to explain the impact of their work.

Research insights

The vast majority of researchers were trying to track the societal impact of their research

Citations were used by three-quarters of respondents to measure societal impact

Citations were also most commonly considered the best way for tracking societal impact

However, this seems at odds with the idea that societal impact is impact ‘outside the sphere of academia’.

Over three-quarters of respondents felt they should be doing more to measure societal impact

Time constraints prevent more tracking of societal impact, but there are other challenges too

Interview insights from academics based in the Netherlands

We carried out a series of interviews with researchers from a variety of disciplines who have made a societal impact with their research.

Researchers agreed that tracking true societal impact was challenging…

I think in any discipline or any area of research, what we should value is contributions to the field. The fundamental problem at the moment is that the metrics used to measure this aren’t really telling the whole story.

Michel Dumontier, distinguished professor of data science at Maastricht University

Read the interview.

Our aim is healthier children – but that's something that's really hard to measure, in the sense of knowing to what extent we’ve contributed to progress. So we’re not measuring that – yet.

Jutka Halberstadt, assistant professor at the Department of Health Sciences, Vrije Universiteit Amsterdam

Read the interview.

I think it’s actually difficult to assess academic impact accurately, let alone societal. With academic impact, we tend to only look at things like publications in high impact factor journals, for example – but does that truly reflect academic impact?

Professor Hans Nelen, key domain chair in criminal law and criminology at the Faculty of Law, Maastricht University

Read the interview.

Collaborations and partnerships were cited as a way to approximately assess impact…

We don't measure [societal impact], as such. But I do look at the number and kinds of collaborations we have as this gives an indication of the reach and impact we can have.

Jutka Halberstadt, assistant professor at the Department of Health Sciences, Vrije Universiteit Amsterdam

Read the interview.

For societal impact, I think one of the key things you can look at in my field is how you’re working with external institutions. Not only in terms of getting the grants, but also in terms of how you work with them when you start new research projects. Your connections with societal institutions – with judges, courts, the public prosecution department, police, defence lawyers, ministries – are a reflection of the impact you’re likely to be able to have.

Professor Hans Nelen, key domain chair in criminal law and criminology at the Faculty of Law, Maastricht University

Read the interview.

[Collaborations] would be an indicator for me that they’re having an impact. I want to know: what programme committees they’re on that have nothing to do with their field; what talks outside their field they’ve been invited to give, whether they're department talks, workshop talks or conference talks; and have they contacted any news organisations to tell them about their research and why it’s interesting.

Michel Dumontier, distinguished professor of data science at Maastricht University

Read the interview.

While others discussed the importance of qualitative measures, self assessment, and being able to tell an ‘impact story’...

For our fellowship, we’ve been discussing the idea that if you don't have enough publication points, you could perhaps compensate by showing your societal impact. The idea was that you would have to tick boxes if you think that you’re outstanding on specific criteria, such as being featured in the media or speaking at conferences. And then you’d have to provide evidence of that.

Professor Frank Cörvers, scientific director at the Maastricht University School of Business and Economics (SBE)

Read the interview.

Publishing research in conferences and journals is an important way to disseminate your work. While the number of citations for published work offers insight into its prominence, it fails to describe why the work matters to others. I ask my researchers not to focus on the number of citations or their H-index, but rather to articulate how others build on their research – whether that’s using the software or data they have created, or in developing new methods and their application.

Michel Dumontier, distinguished professor of data science at Maastricht University

Read the interview.