Have we 'hit a wall' on pushing for broader metrics?

Our Time for change report suggests a slowdown of interest in a broader set of metrics for research evaluation, and a lethargy for change. But is this what we are seeing in the sector? And where should we go from here?

Dr Faith Welch is a Research Impact Manager at the University of Auckland in New Zealand. She has worked in research impact-related roles in the UK and New Zealand, as well as in knowledge exchange and business engagement at one of the seven UK Research Councils.

Steve Lodge is the Head of Services at Emerald Publishing. Emerald has recently launched Impact Services to help the research community plan for impact and ultimately achieve better research outcomes.

Why does the research sector appear less keen on change?

Faith Welch: I think our academic community has reached a period of peak fatigue caused by the ongoing global pandemic. While we have seen increasing support for a broader definition of research excellence, which includes non-academic impact and measures to evaluate this over the past decade or so, nearly two years of uncertainty and restrictions has led to the community collectively ‘hitting the wall’. I think this may have led to stagnation in the drive for change.

Steve Lodge: I agree that fatigue might be taking its toll. The pandemic has stretched resources to the limit and has resulted in so much necessary change that it understandably might affect momentum. In the UK, completing submissions for the Research Excellence Framework was a challenging exercise and while there is wider acknowledgment of the need for more meaningful and inclusive definitions of research excellence, the sector will undoubtedly benefit from greater support to implement real change. That all being said, many of the people I speak to seem extremely motivated to instil practices that will lead to sustained change.

Has the journey to research impact changed during the pandemic?

FW: I think the pandemic has had a really positive impact on impact! It has highlighted the importance of research-informed policy as well as clear and effective science communication. The Wellcome Trust recently found that public trust in scientists rose during the COVID-19 pandemic. On the flip side of this, our academic experts have also been exposed to abuse, threats and harassment due to their public profile. We need to put new measures in place to support the safety of our public-facing experts to ensure their incredibly valuable contributions to society can continue.

SL: I think the pandemic has put a real emphasis on the need for both greater accessibility of research and planning for meaningful outputs. The fact that impact planned through research has become an integral part of funding applications in some quarters, is good progress in itself. However, there does appear to be a lack of agreement on what counts as impact, and this is turn can create a bit of disparity in projects being reviewed and recognised. Moreover, it can create confusion for those conducting the research, who are trying to make a real difference but still require guidance to plan for and articulate impact properly. The journey is as important, if not more important than the end result, so the fact that we are talking about this is still a huge win.

Have you seen a decline in interest for changes to impact metrics?

FW: I think a lot of people put measuring or evidencing impact into the ‘too hard’ basket; it’s just yet another thing for our academics to do. They want a simple list of the best impact metrics and logically think it should exist considering that the impact agenda has been around for years. I’m often met with disgruntled faces when I tell them that it’s not quite as straightforward as that. There are an infinite number of pathways to impact and impacts that could occur, and therefore an infinite number of ways to evidence them. In many circumstances, it is just not possible or feasible to ‘measure’ the impact that has occurred. Instead of impact metrics, I prefer to refer to impact evidence and indicators. I work in partnership with academics to find the unique set of indicators that would work for their specific research project. There is no ‘one size fits all’ when it comes to impact metrics and I don’t believe there ever will be.

I think it is more important to ‘measure’ impact at an institutional level. Here, I think we should focus our efforts on evidencing the environment we have nurtured to enable impact to occur, rather than trying to find a single set of four or five measures to apply to our research portfolio because we will always end up disadvantaging certain areas of research and types of impact. Personally, I like to use the Emerald Impact Healthcheck on an annual basis to indicate progress in successfully implementing our internal impact strategy.

SL: I think people are seeking clarity over something that is seen as quite complicated. You can certainly identify indicators of impact and evidence of change, but it’s extremely difficult and arguably dangerous to seek standardised measurements or simple rankings. Traditionally, research-related metrics have tended towards absolutes or easy proxies, such as using the impact factor or prestige of a journal to indicate an articles’ quality. This has – and still often does – bias people against publishing in certain journals, and researchers are still rewarded for publishing in ‘better’ journals. Yet journals intentionally designed to communicate to practitioners, educators and broader non-academic communities may be far more effective in reaching the right audience. So, as we battle with the oversimplification of metrics across the research sector as a whole, we need to ask questions about the most appropriate way to drive, showcase and measure impact.

Are activities around impact literacy & impact readiness gaining traction?

FW: Since I started in my role as Research Impact Manager at the University of Auckland in 2018, I have seen a constant increase in the demand for research impact training. A lot of this has been driven by increasing emphasis on impact from our research funders. In 2020 and 2021 we saw various COVID-specific funding calls, all with impact making up a significant proportion of the assessment criteria. I expect the urgency required to tackle global challenges will continue to drive an increasing level of impact literacy worldwide.

SL: Impact is certainly gaining traction internationally. We’ve noticed both growing interest in impact globally and increasing investment in building impact literacy particularly in the UK, Australia and New Zealand. People are reaching out to us and our partners for guidance and training, and impact literacy is starting to feature not only in individual development but also the processes of funding and assessment. It’s wholly reassuring to see impact not only growing, but healthy approaches to impact gaining ground.

Where should the sector focus its attention to further research impact?

FW: While still important, I think the industry spends far too much time discussing impact measurement. Let’s instead focus on creating the best possible environment for impact to thrive. The industry should be working together to break down the barriers to achieving impact and ensure that we are equitable in our approach to doing so to embrace the diverse range of impacts research can have.

SL: If the industry collectively agrees that research should be leading to real change then we need to ensure that researchers and research institutions who are instilling this culture of impact are recognised for their efforts. We also need to ensure impact is embedded from the outset, through realistic, achievable plans, and undertaken in collaboration with stakeholders at all stages.

How can the industry further change?

FW: Ways we could collectively further progress include:

• Creating a sustainable model to make research outputs accessible and free to everyone

• Sharing what does and doesn’t work. The University of Auckland launched a webinar series in 2020 called Impact through Culture Change to do exactly this and we hope to continue to play our small part in sharing best practice in 2022

• Understanding how we can protect our academics from abuse and harassment so they can continue their invaluable responses to critical global challenges such as the pandemic and climate change

• Celebrating the work of everyone who helps enable research impact! The researchers, the librarians, the impact specialists, the knowledge mobilisers, the community engagers, the commercialisation experts, the communications teams, and so on.

SL: Rewarding and recognising impact beyond traditional measures is vital, but demands a fundamental shift in academic reward systems. Moves to drive greater collaboration with stakeholders are similarly essential, but likely also require investment in technologies or staff capacity to make coproduction easier. From a UK perspective, focus will likely narrow to the next window of assessment once the results of the 2021 REF have been announced, so there is a really important opportunity now to address the environments within which impact can thrive. I completely agree with Faith, impact doesn’t happen in isolation – let’s recognise the work of all the people involved!