How can we fairly evaluate research?

Authors: Emerald’s Impact Advisory Board. 

Research metrics such as journal impact factors, citation counts and h-indices are used to evaluate research impact and quality. Research metrics play a role in the assessment and management of research, but the story they tell is limited. Used in isolation or in the wrong context, metrics can unfairly disadvantage some researchers.

Many across the sector are calling for more responsible use of metrics, broader evaluation measures and reforms to research assessment practices in general. There have been numerous efforts to change the system, notably the San Francisco Declaration on Research Assessment (DORA), The Leiden Manifesto, The Metric Tide, and most recently the Agreement on Reforming Research Assessment. Many research institutes have released statements on the responsible use of metrics, and funding bodies, publishers and organisations have made commitments to work together for change.  

Commitment for change

Emerald is among those championing alternatives to traditional academic metrics and rewards. It is one of six commitments we have made as part of our Real Impact Manifesto and Are you in? movement. Part of our role is to encourage the research community to share experiences, ideas and best practices for change. We believe that by working together we can create a fairer and more inclusive system that rewards and supports quality and impactful research for a better world.

Our manifesto was developed in partnership with our Impact Advisory Board and reflects their views on the barriers to change. Building on this work, we had a discussion with the Board to explore the current challenges and opportunities around alternatives to traditional academic metrics and rewards.

---

Shifting sands

Ken Knight is a Research Impact Manager at Murdoch Children's Research Institute in Australia.

Many of the challenges for broader measures of impact centre around capacity for change, says Ken. In his view, institutions may commit to change and sign initiatives like DORA, but they don’t necessarily have the capacity to develop a strategy they can deliver on to address the challenges.

Ken sees academic culture as another blocker to change, particularly for researchers in the health and medical context who often have a deep commitment to traditional metrics and performance measures. "Very senior researchers and clinicians are comfortable with the metrics they’ve used their entire career," he says. "And they have very well-established narratives around those metrics and ways of thinking about performance."

"But I also think the systems that capture those metrics and indicators are the ones that are most developed and the ones we already have access to. So, while there are external drivers for change, there aren’t enough, and certainly none that are compelling or coordinated."

Another major barrier to change lies in the precarity of the academic environment, says Ken. In his view, researchers are less willing to challenge the status quo because of job insecurities and increased competition for funding. "We already exist in a hyper competitive and brutal research funding environment, and I think people feel an appropriate level of anxiety and unease around metrics and evaluation," Ken says. "I think the elephant in the room is about ‘Is this going to make me look bad?’ and ‘Am I going to jeopardise whatever tenuous funding I have, if people are going to be evaluating the value of my research in new ways’."

The outlook for change isn’t all gloomy though, notes Ken, pointing to growing interest in initiatives like the Impact CV which was created by Australian researcher Katherine Andrews. Institutions are also beginning to explore new ways to benchmark and evaluate performance, says Ken. "I would like to see research engagement, co-production of research, partnerships, translation and implementation activities, and processes and planning factored into academic promotion criteria," he adds. "I think that's something that really needs to be addressed at the university level."

Metrics and academic careers

Louis Coiffait-Gunn is Director of Policy and Public Affairs at The Publishers Association.

One of the main challenges that Louis sees with current research evaluation systems is the use of citations and JIFs in hiring and promotion. He would like academia to try anonymous recruitment as other sectors are starting to, and to consult a broader range of metrics, including Altmetric and Overton.

Louis believes publishers can play a key role in helping facilitate relationships for researchers with policymakers, businesses and community groups. He also thinks publishers and communication experts could work with academics to create new formats for communicating their research, to reach new audiences. "So much research is still locked away in PDFs and very technical language" says Louis. "I'd love to see teams of different communication experts take ‘raw’ research material and work with academics from the start on all kinds of multimedia outputs, for different audiences and purposes."

More than a number

Lara Skelly is the Open Research Manager for Data and Methods at Loughborough University, UK.

Issues around metrics often relate to how they are used in performance evaluation, says Lara, who argues for change within HR practices. "Academia is supposed to be at the forefront of these things, so why are our HR departments still working with outdated, outmoded HR practices, and applying it to people who are creative, who are bigger than a number?" she asks. "I wonder if the push isn't with academics or even in management, maybe it's just as simple as an HR practice."

Another challenge for Lara is around academia’s mentorship model. She believes that because supervisors pass on their traditions and culture to their students, old ways become entrenched. "I think the mentorship model has great benefits, I'm not discounting them, but it poses a problem if we are going to change," she adds.

In terms of communicating science, Lara sees a need for a broader range of formats, including ways to present experience as a research output.

Looking at positive steps towards new ways to measure impact, Lara has been impressed with EFMD's Business School Impact System, an impact assessment tool for business schools. "I like the way that they are currently thinking about impact, although I wonder if it goes far enough. They are constantly updating their views and providing guidance to business schools. It’s something I keep an eye on," she says.

Impact through lived experience

Sharon Zivkovic is Adjunct Research Associate at the University of South Australia, Founder and CEO of Community Capacity Builders and the Cofounder and Chief Innovation Officer at Wicked Lab.

New approaches to social enterprise and autism research could help shape thinking around impact, says Sharon. Lived experience is one approach that is attracting attention in autism research, and it is already being applied in peer review for some journals. "There are a lot of movements starting to happen who are a bit anti the researchers, particularly in the autism community," explains Sharon.

The Global Autistic Task Force on Autism Research have been writing into journals as a right of reply and the lived experience researcher kind of movements are happening around mental health and autism. This is where people have been disenfranchised by researchers and are starting to organise, which is quite exciting to have their point of view put across in academic journals as well."

Turning to ways that researchers can advance the impact of their research, Sharon believes more should be done to influence decisions at the policy level. Policy briefs are a useful tool, says Sharon, because they communicate research quickly and don’t rely on research papers being published. Sharon frequently writes policy papers and has recently submitted one on the creation of autistic-led social enterprises to feed into the development of a national employment framework for autism in Australia. "To get policy change, it's got to happen quick, you can't wait to publish a paper or anything like," she adds.

For researchers who need support in communicating their research, Sharon is a fan of initiatives such as Fresh Science in Australia (managed by Science in Public) who run a national competition and training programme that helps researchers communicate their findings. "Fresh Science gives awards to early career researchers from different universities in Australia and the final night, you've got to get up on stage and talk about your research in the time it takes for a sparkler to go out," she says.