Replication studies: learning from failure and success

Short commentary illustrating the importance of replication through two examples

Anne-Wil Harzing - Sat 9 Jul 2016 08:17 (updated Fri 3 Jun 2022 08:19)

My co-author Arjen van Witteloostuijn is on a mission to change the way we do research in the Social Sciences. After a petition on Change.org that you can still sign, his latest effort is an article in Cross Cultural & Strategic Management, which, with its new editorial team, looks like becoming one of the most provocative and open-minded International Business journal. I was invited to write a commentary on this paper, in which I focused on replication studies.

• Harzing, A.W. (2016) Why replication studies are essential: learning from failure and success, Cross Cultural & Strategic Management, vol. 23, no. 4, pp. 563-568. Available online - Publisher's version

What happened to Popperian Falsification?

Van Witteloostuijn’s (2016) commentary “What happened to Popperian Falsification?” is an excellent summary of the many problems that plague research in the (Social) Sciences in general and (International) Business & Management in particular. As van Witteloostuijn (2016:pp] admits his “[...] diagnosis is anything but new – quite the contrary”, nor is it applicable only to the Social Sciences. When preparing this note, I was reminded of Cargo Cult Science, a 1974 Caltech commencement address by Physicist Richard Feynman (Feynman, 1974), which – more than four decades ago – makes many of the same points, including the pervasive problem of a lack of replication studies, which will be the topic of this short rejoinder.

Replication studies in International Business

Conducting replication studies is more difficult in International Business (IB) than it is in many other disciplines. For instance in Psychology – a discipline that favours experimental research – one might be able to replicate a particular study within weeks or, in some cases, even days. However, in IB data collection is typically very time-consuming and fraught with many problems not encountered in purely domestic research (for a summary see Harzing, Reiche & Pudelko, 2013). Moreover, most journals in our field only publish articles with novel research findings and a strong theoretical contribution, and are thus not open to replication studies. To date, most studies in IB are therefore unique and are never replicated. This is regrettable, because even though difficult, replication is even more essential in IB than it is in domestic studies, because differences in cultural and institutional environments might limit generalization from studies conducted in a single home or host country.

Replication failure and replication success

Somehow though, pleas for replication studies – however well articulated and however often repeated – seem to be falling on deaf ears. Academics are only human, and many humans learn best from personal stories and examples, especially if they evoke vivid emotions or associations. Hence, in this note, instead of providing yet another essayistic plea for replication, I will attempt to argue “by example”. I present two short case studies from my own research: one in which the lack of replication resulted in the creation of myths, and another in which judicious replication strengthened arguments for a new – less biased – measure of research performance. Finally, I will provide a recommendation on how to move forward that can be implemented immediately without the need for a complete overhaul of our current system of research dissemination.

Want to publish replication studies?

Here is a list of journals in Business & Management that are known to publish replication studies. If you are aware of other journals that do so, please drop me a line.

• Cross Cultural and Strategic Management
• Management Review Quarterly
• Strategic Management Journal

References

Feynman, R. P. (1974). Cargo Cult Science. Engineering and Science, vol. 37, no. 7, pp. 10-13. Available online - Wikipedia article about Cargo cult science

Harzing, A.W.; Reiche B.S.; Pudelko, M. (2013) Challenges in international survey research: A review with illustrations and suggested solutions for best practice, European Journal of International Management, vol. 7, no. 1, pp. 112-134. Available online - Publisher's version (free access!)

Witteloostuijn, A. van (2016). What happened to Popperian falsification? Publishing neutral and negative findings: moving away from biased publication practices. Cross Cultural & Strategic Management, vol. 23, no. 3, pp. 481-508.

Related blogposts

• Should we distance ourselves from the cultural distance concept?
• Challenges in International survey research: illustrations and solutions
• Replication study gives thumbs up for the individual annual h-index
• Are referencing errors undermining our scholarship and credibility?