Research evaluation

Losing our appetite for change

The apparent gains towards a broader set of metrics for research evaluation and general appetite for change appear to be slowing down. In this year’s survey, 47% of academics want to introduce metrics beyond citation metrics, just up on 46% in 2020, but still down on 51% in 2019. There is slightly more interest in dropping JIFs altogether – 18% in 2021, up from 17% in 2020 and 10% in 2019. However, less than a third (29%) want to change the way incentives are used to publish research, a steady drop from 30% in 2020 and 32% in 2019.

There are wide differences of opinion at the regional level. The UK is most keen to drop citation metrics (23%) and least interested is Latin America (6%). Enthusiasm for introducing other metrics beyond citation metrics is highest in SSA (60%) and Latin America (53%) and lowest in Australasia (43%) and UK (42%). Latin American and N&WE (excluding UK) are most eager to change the way incentives are used to publish research, with 36% in each region choosing this option.

A return to metrics?

For the second year running, participants chose journal citations and impact factors as the main way research impact is currently measured – with 70% opting for this, compared to 58% in 2019 (71% in 2020). This option is most popular in S&EE at 79% and lowest in the UK at 60%, both consistent with 2020. Other areas to rank high are tenure or career advancement (32%) and funding opportunities (26%) and provable effects of research in the real world (26%).

Mobilised knowledge that affects decision making is at the bottom of the choices globally (18%) with SSA the only region that had more than 1 in 4. There continues to be less focus on measurable changes in practice, policy, or behaviour, standing at 25% in 2021, slightly up from 2020 (24%) and considerably down on 2019 (59%). However, UK interest in this measure of research quality has risen to 45% from 42% in 2020.

Despite the need for research to influence real change in response to the pandemic, improved societal health, economic or environmental outcomes as measures of research impact continues to remain relatively low – standing at 21% in 2021 and 19% in 2020, compared to 61% in 2019.

Waning support for new impact measures

Academics remain less keen about changing the way research impact is measured than they did in 2019. 34% of global respondents in 2021 and 33% in 2020, compared to 38% in 2019 say they are very open to change and have already driven change. Academics believe their institutions are even less supportive of change than they are personally, with figures for very open to change and already driven change dropping from 16% in 2019 to 13% in 2020 and 2021.

In 2020, we found that SSA, ME&NA and India compared to other regions are more likely to think that people in their institutions had already driven change. However, this year India is the outlier with nearly 1 in 4 believing this and the other two regions falling. It is still the case that Europe, North America, UK, and Australasia are less likely to agree that change has been delivered but especially those in N&WE (excluding UK), where only 4% agree compared to 11% in 2020.

Main challenges to change

The top three barriers to change are career incentives (55%), difficulty of tracking impact beyond academia (51%) and lack of clarity on what other measures could replace rankings (49%). There is an increase in organisational resistance to change and entrenched culture – 43% in 2021, compared to 34% in 2019 – although slightly down from 2020 (43%). Lack of funding for open research appears to be more of a challenge in 2021 – 41% compared to 39% in 2020 and 26% in 2019.

At the regional level, career incentives still aligned to traditional impact metrics is most popular among those in Australasia (70%) and North America (65%).

Unlike in 2020, the UK isn’t top in any challenges and features nearer to the average. This year, North America scores the highest for organisation resistant to change (61%), lack of clarity on what measures would replace rankings (63%) and difficulty in tracking research impact beyond academia (66%).

There is a significant difference of opinion between genders when it comes to incentives for career progression still aligned to traditional impact metrics, with 66% of women, compared to 51% of men choosing this option – the difference has slightly increased since 2020.

What are academics prepared to do?

To drive change, 46% academics globally are willing to publish open access and share links to supporting datasets – this was also the top choice in 2020 (51%). Next popular is publishing non-traditional content if the rewards mechanisms were in place (43%). The low- or middle-income regions are least attracted to publishing non-traditional content with Europeans, North Americans and Australasians choosing this around two-thirds of the time. There are also different levels of support among the genders for publishing non-traditional content – male academics are less keen (40%) than female (51%).

Changing their processes/methodologies is the option least favoured by academics, with only 13% globally backing this option.

There is a significant difference of opinion between genders when it comes to incentives for career progression still aligned to traditional impact metrics, with 66% of women, compared to 51% of men choosing this option – the difference has slightly increased since 2020.

What will drive change?

Academics believe that more opportunities for collaboration with industry and practice is key for driving change, with 60% choosing this option. Next important is having more publishers make research open access (50%). A third are in favour of having more debate of the issues in a public forum, followed closely by impact literacy training (31%).

There are different levels of support for the options depending on academics’ career stage. Students are particularly in support of driving change through impact literacy training (42%), collaboration between industry and practice (73%), and communications support (27%). Five of the six highest chosen options are from the early career stages, with only greater adherence to DORA principles (19%) gaining more support from those in the mid-career stage of 16-20 years post PhD.

The data

How important is demonstrating impact of research on society to...?

How is the quality of your research impact currently measured?

How important are the following factors in helping achieve broader impact with your work?

How strongly do you support the idea of changing the way research impact is measured?

How supportive/interested are those in your broader institution in driving change when it comes to other ways to measure research impact?

What main change would you like to see in the way research quality is measured?

Which of the following do you consider to be the biggest ‘challenges’ of changing the way research impact is assessed??

When it comes to your research, what types of change would you consider implementing??

In your opinion what are the best way(s) to enable change to happen? (% of times chosen in top 3)