Alternative indicators and the qualities of research

Author: Rebecca Evans, Product Manager (Open Access), Emerald Publishing

It is not revolutionary to acknowledge that scholarly publication metrics are used inconsistently, with limited bibliometric data (that of journal impact factors, citations, and h-indexes) serving as a proxy for the quality of the article. Articles are not considered holistically or evaluated on their specific aims. But some researchers may be locked out of these high-impact factor journals, or may not receive citations for a myriad of reasons, whether these are institutional, geographic, or related to their career stage or research focus.  

The Open Lab at Emerald Publishing has argued before that open access itself does not automatically engender equity, and our hypothesis here is that ensuring equity in open access alone is not enough; we must ensure equity in participation in the scholarly conversation as a whole. Would alternative indicators help break down barriers by reimagining success?

We considered creating a broader basket of article-level indicators that would be more inclusive, relevant, and specific to researcher aims. The four themes would be:

• Scholarly attention (research is contributing to scholarly conversations)
• Engagement (research is contribution to conversations outside of the scholarly community)
• Change (real-world changes have been anticipated, are planned or are happening as a result of the research)
• Transparency (authors have acted with transparency and the article follows recommended ethical practices)

Each theme would be measured with four binary questions covering a mix of qualitative and quantitative aspects, with the former benched against subject averages. This would be underpinned by providing a dynamic narrative context for the values that updates over time, and education how to improve areas in the future depending on the researcher’s aims. Different articles have different qualities and researchers have different priorities. Our purpose isn’t to judge or rank values, but to help form a holistic picture of an article that can serve both authors and readers by signposting applications and values of a work.  
 
Emerald has been a signatory of DORA since 2019, and the Alternative Indicators exist within the context of the latter; it is a way to help ‘ assess research on its own merits’ (Read the Declaration | DORA), by making a range of article-level metrics available (although we hesitate to consider the indicators as ‘metrics’ as this is not intended as a measurement or assessment exercise).  

Perhaps the more recent context to consider the Alternative Indicators in is the newly released ‘The State of Research Assessment’ white paper, published by Springer in April (The state of research assessment).

The report advised that quantitative methods are used for 93% of researchers in evaluating their work. Researchers want to continue to be assessed on their research outputs, but to also have an increasing focus on contribution to, among other points, the public good. Some of the examples in which interest was demonstrated relate directly to the topics we were looking to cover within the indicators, for instance:  

• Benefits global challenges (Change)
• Innovation activity (e.g. patents) (Change)
• Influence on policy (Change)
• Industry collaborations (Engagement)
• Co-creation with communities (Engagement)

It was heartening then that our Alternative Indicators proposal appeared to be in step with how researchers looked to be evaluated – ultimately looking for a more holistic evaluation of their work.  

We put the idea to different stakeholders within academia. And it was generally well-received, with interest in bringing the various qualities of an article to the forefront. Some were very enthusiastic. There were some negative opinions among the voices, fairly expressing that applying values will result in it be being measured against, regardless of intent. This needs careful consideration by any who adopt the use of indicators and metrics for any form of evaluation. One size cannot fit all and research cannot be ‘measured’ in the same way, if at all. 

The greater question was around the ‘Change’ bucket and whether this vital research outcome can truly be measured. It is difficult enough to demonstrate or articulate intent to change (e.g. through SDG alignment, having an impact statement/plan, or linking to a real-world call for research), and continued efforts are needed to help drive research impact awareness. Determining where actual change had occurred due to the research on which an article is based is even more difficult. It also often occurs on a longer time-scale than many of the other indicators. Springer’s report speaks to this too: it is objectively more difficult to measure benefit to the public good.

Should this area be changed to something more immately measured using publisher data? I personally would be hesitant to do so; this is the type of information we need to start surfacing and presenting, particularly given the threat research as a whole is currently under. At Emerald, we are committed to research with real-world impact, and the Open Lab’s choice to focus on change speaks to that. But it requires more thought for the moment. Presenting wider research attention and engagement is hugely beneficial, but we believe greater efforts need placing on incentivising research impact, as well as surfacing ‘change’.  

Alternative Indicators have been paused for now, as more work is needed before we can consider how to apply our thinking. We will continue to assess how we can best present valuable research qualities beyond traditional metrics and if you would like to get in touch to express your thoughts either on Alternative Indicators, or anything else we are working on, please get in touch at: [email protected].