Do top journals stop creative research?

Dr Gangaram Singh describes his studies of the role of publication in top-rated journals in research and researcher merit assessment, and the challenges he faces in his new role at one of the largest US universities.

---

What prompted your interest in management and management journals?

Prior to beginning my role as Associate Provost at National University almost a year ago, I served in many roles in the Business School at San Diego State University, from 1999 when I joined as an Assistant Professor. Later, I chaired the Management Department for four years, then moved into the Associate Dean's role, where I championed the reaffirmation of The Association to Advance Collegiate Schools of Business (AACSB) accreditation.

My PhD is in Industrial Relations (IR) from the University of Toronto. Like many IR scholars, I ended up in a business school, so management became my home discipline. This traditional route meant I was going to be judged by my publications in management journals, but deep down I was trained to publish in excellent quality IR journals – the iconic journals in IR (such as ILR Review) were relegated to second-class status. I often wondered if going down that path was the right choice. 

Your research into "top" management journal articles has been very influential. What motivated it?

I am the product of an international education. Hence, I believe that we can learn from a variety of perspectives, and it is inherently incorrect to restrict knowledge to a particular view or source. 

Traditional, so-called "top" journals did just that; restricted the creation of knowledge to a formula. Moreover, top journals often prescribe/imply parameters for rejection. I would argue that too often an idea and/or its contribution to knowledge/decisions is thrown out according to the criteria for rejection.

What do you mean by a "top" management journal?

Many criteria are proposed for defining a "top" journal. For example, it could be the collective impact of all its articles (journal impact factor), or rejection rate, circulation or how senior researchers in the field would rate it. Primarily, however, the impact score for the journal as a whole is considered the Holy Grail.

Top journal rankings are very important to universities because they can be the source of funding (particularly in Europe), bragging rights for a particular institution, and even recruitment of students into a school. The major reason, though, is for the tenure and promotion of faculty. Some schools even go as far to say that a faculty member must publish x articles over y years in z journal. 

It was important to document the logic that "top" journals could publish less impactful articles, and "non-top" journals could publish impactful articles. So I joined forces with some like-minded scholars, and brought the evidence to light.

Recently, you investigated the influence top journals have on marketing. Can you outline your findings?

We examined the Type I and Type II errors for classifying top articles in the sample, and first demonstrated the extent of the misclassification in management. We then replicated the study in accounting and finance, and most recently marketing. As expected, we found similar results. In fact, I am sure if we replicate our methodology in non-business disciplines, we will also see a significant level of misclassification.  

What are the potential implications of your research? 

Public policy that uses journal rankings for funding should take note. Universities that use journal rankings for selection, development and promotion of faculty should take extreme note. Scientific rigour with no practical substance or impact can lead to irrelevance.

Do you plan to extend this research?

No, I think the point is made, and it is up to the community to test its usefulness. I am pleased that it has received so much recognition (impact!). And clearly, my findings were not published in the so-called "top" list. I only hope that our thoughts will be impactful in the selection, development and promotion of faculty who bring more than scientific rigour to higher education. 

How do you see your work evolving over the next few years?

I am excited to join National University, a private non-profit university with a mission to provide higher education opportunities to working adults. National University has embraced the non-traditional approach with qualified access, online, face-to-face and hybrid modalities, accelerated learning (one course per month), and rigorous assessment as its core strategy.  I now need to turn my attention to meaningful practical research on how we can provide more opportunities for adult learners to complete their higher education journey. I may even bring some of my traditional scientific researcher skills to this endeavour!

---

Exclusive clubs: how top journals block intellectual creativity

National University and San Diego State University researchers have gathered compelling evidence that publication in top-ranked management or business journals is no guarantee of article quality and impact.

Scientific journals provide a vital outlet for research dissemination and transferring ground-breaking knowledge into new practices; and, inevitably, some journals are ranked as more prestigious than others. Most journal ranking systems use citation metrics as article impact indicators from sources such as the ISI Web of Science and Google Scholar. Though there are many highly respected journals published locally, English-language journals in the business domain carry most cachet. Their refereeing and peer evaluation effectiveness, impact ratings per article and, especially, restricted access influence the perceptions that drive journal prestige rankings. Publishing opportunities in the journals ranked as top-quality are limited, so competition for publication in them is fierce and acceptance rates are low.

National and international academic institution rankings increasingly depend on their perceived research output quantity and quality – external observers assume that a high number of research papers indicates productivity, and that high citation metrics reflect high-impact work. Publication in top journals thus elevates institution status, which institutions may then use to attract research funding, corporations interested in collaborations, superior staff and talented students. Some faculties even use top journal publication history as an evaluation tool for member selection and reward, according to Dr Gangaram Singh, of the National University in La Jolla, California. Corollary to this, personnel selection committees tend to automatically downgrade articles published elsewhere than in the top journals, and some go as far as to disregard them outright.

Singh and colleagues from San Diego State University explored the extent to which top-ranked management and business journal publication signalled article quality in studies between 2007 and 2014. They found that articles, whether research-, case-, education- or applied science-oriented, were consistently cited more often if published in top-tier journals.

Journal rank as proxy

Singh and his colleagues carried out assessments of journal ranking importance in inferred article quality in the fields of management, accounting and, in their latest study, marketing. The studies showed that, even among these disparate scientific communities, the populations of top journal classification for a particular field tend to remain static, though their individual order may change.

While studying management articles, they established a baseline of the top five journals: Academy of Management Journal; Academy of Management Review; Administrative Science Quarterly; Journal of Applied Psychology; and Strategic Management Journal. They then rated the impact of particular articles first published in 1993 or 1996, and also assessed the source of the citations to measure each article's impact on the work of others.

Singh and his colleagues found that inclusion in any one of the five journals was sufficient to lead to more citations. However, applying three criteria to placement of articles into top and non-top categories, using journal ranking as a proxy for quality, delivered erroneous results. The first two criteria were the median and mean numbers of "pure" (non-self) citations for articles in each focal year over seven years; the third was the number of articles published in the top five journals in each focal year as a proportion of all articles published in the field's 34 journals.

They found that self-citation was prevalent in certain journals – one journal's score for articles published in 1993 was 56 per cent – but that even allowing for self-citations, the results did not much vary. Ultimately, the study found that the five top journals did proportionally issue more top articles. However, 63 per cent of the top articles were published in other journals. 11 per cent of the articles attributed top status did not meet the study's top status criteria and 25 per cent of the 241 articles in the top journals did not merit top article status once their inclusion in the top journals was removed as a determining factor. And even applying top journal inclusion as the main selection factor, 53 per cent of the top articles published in 1996 were published by non-top journals.

Read the article

Following a similar methodology, Singh and his colleagues explored top article/top journal relationships in accounting and, later, marketing. For accounting, using the Social Science Citation Index, the top journals were: Journal of Accounting and Economics; Journal of Accounting Research; and The Accounting Review. For marketing, they determined the citations for all articles published in the key 26 journals in the Social Science Citation Index, and 50 in Google Scholar, compiling a top four: Journal of Marketing; Journal of Marketing Research; Journal of Consumer Research; and Marketing Science.

The results of both studies also showed that top article status cannot be inferred reliably from inclusion in top-ranking journals: "Management and business schools are faced with an ongoing crisis – the desire to fit applied research into the scientific paradigm. But such research may result in an obscure or small, insignificant idea heavily supported with scientific rigour," observes Singh.  "As we embrace a culture of evidence-based decisions, the article should be the unit of analysis for impact, not the journal."

The reason for increased academic emphasis on top journal publication record partly lies in ever-growing specialization, which means that even academics from the same field may not be able to determine an article's merits. Additionally, as available top journal article space is relatively small compared with the number of contenders for publication, the top journals act as gatekeepers, prioritizing thoroughness and scientific rigour over innovation. Scientific communities that value rigour thus value the journals' contributions. Singh's view is that this scientific gatekeeping leads to intellectual stagnation and disconnection from the needs of the practicing business community: "Traditionalists invariably come to the rescue of top journals, especially if they have published in one of them," he muses.  

In Singh's view, using journal reputation as a determinant of research quality is not only mechanistic and naive, but also confers unfair advantage to undemanding research. Some of the best research will never make it into the top journals, simply because of its challenging or unpolished nature. Whether excellent research will receive the attention it deserves is too important to leave to the scientific publishing elite. Researchers with something meaningful to say, whose work will add value and contribute positive change, should be undeterred and publish where they can; likewise, policymakers and institutions should ensure that they read an article before they make decisions about whether to include its message, or author, in their club: "I will continue to read each article, and not abdicate responsibility," Singh says. "Get creative, relevant ideas out into the research marketplace! Allow them to spark relevance."

Whether excellent research will receive the attention it deserves is too important to leave to a scientific publishing elite. Researchers with something meaningful to say, whose work will add value and contribute positive change, should be undeterred and publish where they can.

---

Unravelling the complexities of journal impact in business management, finance and marketing