Australian research output in Economics & Business: quantity over quality?

Shows the discrepancy between research productivity and research impact for Australian research in Economics & Business

Anne-Wil Harzing - Mon 1 May 2017 09:07 (updated Thu 2 Jun 2022 17:15)

This paper – written during my first sabbatical leave at the University of Melbourne in 2004 – marked the start of my interest in research evaluation and research impact. Since then, and especially since the launch of the Publish or Perish software in 2006, I have developed a substantive research programme in this area.

Australia ranked 4^th ... or 19^th

Using the Thomson Reuters Essential Science Indicators – which ranks the top 1% most cited universities world-wide in each field – I noticed that Australian research output (i.e. the number of published publications) in Economics and Business was very high for the size of its population. Australia ranked 4^th ex-aequo with Germany, being outranked only by the USA, the UK and Canada, all (much) bigger economies. Australia in turn outranked much bigger European economies such as France, Italy, and Spain.

However, the story for impact (citations per paper) was quite different; Australia ranked only 19^th in the world on this measure and was outranked not just by the USA, the UK, and Canada, but also by the aforementioned Southern European countries, as well as most small European countries such the Nordic countries, Austria, Switzerland, the Netherlands, and Belgium.

Interestingly, it turns out that this mismatch in ranking for productivity and impact was not a temporary fluke. When I verified Australia’s rankings for Economics and Business in the Essential Science Indicators ranking in September 2016, i.e. more than twelve years after the original data collection, I found them to be identical, i.e. 4^th for the number of publications and 19^th for citations per paper. So it seems the phenomenon is persistent. New Zealand outranks Australia at #16, a ranking that is higher than its ranking for publications (#24).

Looking at individual universities paints this picture in even starker relief. As the table below shows, the top-3 universities in Economics & Business in Australia have further improved their world ranking in terms of the number of publications. However, their ranking in terms of citations per paper has slipped to the lowest regions of the Essential Science Indicators (there are currently 270 universities ranked for Economics & Business). When I updated my analysis in July 2018 (see below), the story hadn't changed at all. There was a further improvement in their world ranking in terms of the number of publications, but their ranking in terms of citations per paper continues to languish in the lowest regions of the ranking (there are currently 313 universities ranked for Economics & Business).

Update July 2018: an Australian "productivity boom"?
... or maybe just a database expansion?

A "good news" story about the "productivity boom" of Australian researchers in the 18 July issue of University World News prompted me to update this analysis once again.

The University World News story is based on an analysis conducted by "academics at the Innovative Research Universities (IRU) group". In their report, they compared Web of Science publications for 2016 with publications for 2006 and concluded that Australia had experienced a very high level of increased productivity, with publication output more than doubling in a decade. This was true even though the number of academics in Australia had only increased by a third over the same time, thus leading to the claim of a "productivity boom".

Although the study does acknowledge that China's increase in productivity far outstrips that of Australia, its main conclusion appears to be that Australia has performed much better than the UK and the USA, which only showed increases of 49% and 30% respectively. A similarly postive story is told about citations. Whilst I do not doubt the publication and citation figures are correct, I think it is a mistake to attribute them solely to higher levels of productivity and impact, let alone draw the conclusion that these positive developments are to be attributed to Australia's research funding system.  

In order to illustrate this point, I will compare Australia's changing performance more systematically with that of other countries. Doing a country-by-country comparison for individual years is very time-consuming in the Web of Science interface. To replicate the report's analysis, I therefore used Clarivate's Web of Science "Essential Science Indicators" which ranks countries (as well as universities and individuals) on the number of publications and citations, an approach I also followed for the original paper on which this blogpost is based. I compared the 30-odd "most-publishing countries", using my original 1994-2004 data on the one hand and the most recent data-set (i.e. 2008-2018) on the other hand. Although this comparison is not identical to a comparison between 2016 and 2006 only, the results will be quite similar as publication and citation trends tend to change only slowly. My comparison showed that Australia had indeed quite dramatically increased both its number of publications and its number of citations, but so had most other countries.

Australia had increased its number of publications by 147% (i.e. even more than the 112% that was reported for the 2006 to 2016 period); the UK (56%) and the USA (43%) indeed showed only modest increases, "beating" only Russia, the Ukraine and Japan. However, beyond China there were eight other countries that showed a level of increase in overall productivity that was higher than Australia's: Turkey, Brazil, South Korea, Singapore, India, Ireland, South Africa, Taiwan and Poland. Moreover, there was a wide range of Western countries showing levels of increased productivity that were very similar to Australia: Spain, Norway, Greece, Denmark, New Zealand, Belgium, Austria, Switzerland, Italy and the Netherlands. With regard to the increase in the number of citations, there were even more countries that outperformed Australia.

Database expansion: regional journals

However, much of the increase for all countries involved can likely be attributed to an expansion of the Web of Science database. Over the past decade the Web of Science has substantially increased its coverage of "regional" (regional being "WoS speak" for Asia-Pacific, European, Latin-American, and Middle-Eastern & African journals) journals, books and book chapters. In 2008 alone, the Web of Science added 700 new regional journals. No less than 52 of these journals were Australian, making Australia the country with the largest number of journals added to the Web of Science database. The only other countries that saw a significant number of their journals added were Germany (50), Poland (43) and Spain (43), all much larger countries in terms of their population.

In Economics and Business, the discipline studied in the paper this blogpost is based on, Australia's relative increase in publications was even less spectacular: there are no less than fourteen countries with a more subtantial increase and another half dozen with a very similar increase. The picture for citations is identical. This is all the more suprising as no less than a third of the journals added in Economics & Business were Australian journals. Thus one could argue that we could have expected a much more impressive increase in "productivity" for Australia.

No change in Australia's worlwide ranking

As a result, even though Australia dramatically increased its number of publications and citations covered in the Web of Science, there is very little movement in the relative ranking of countries, neither by number of publications nor by citations per paper. No less than 14 years after the original data collection, Australia still ranks #11 for the number of publications in all disciplines combined, and - just like in 2004 - it is narrowly beaten by Spain at #10. The expansion of coverage and developments in publishing behaviour in emerging economies, however, have catapulted China from rank #9 to rank #2. Moreover, India now outranks Australia (moving from #13 to #9) and South Korea and Brazil have likewise improved their ranking (from #16 and #19 to #12 and #13). South Korea might wel eclipse Australia soon, having only 4% fewer publications in the WoS. In terms of citations per paper, Australia's rank likewise has not moved an inch: it is still at #17. Virtually all developed Western countries, as well as Singapore, have higher impact scores than Australia. The only countries in this group that Australia leaves behind are Italy, New Zealand, Spain, Greece and Portugal.

For Economics & Business, Australia has slightly improved its rank in terms of publications as it is now ranked a solid #4 rather than a shared #4. However, this is mainly caused by Canada dropping to #5 as Germany, which was originally ranked ex-equo with Australia, has increased its publications more than Australia and is now ranked #3. In terms of citations per paper Australia's rank has also slightly improved, now standing at #18 instead of at #19. However, Australia still has one of the largest productivity/impact gaps, only China and Japan do worse in this respect. In terms of citations per paper, Australia was outranked not just by the USA, the UK, and Canada, but also by most European countries (only Italy, Portugul, Spain, and Greece do worse), Singapore, Israel and New Zealand. The latter improved its citations per paper ranking from #18 to #15 and thus increased its lead on its bigger neighbour.

Celebrate, but celebrate cautiously

By all means celebrate the fact that Australia continues continues to pull well above its population weight in terms of publications and citations. Also celebrate that on a per capita basis Australia has slightly improved its world ranking for both publications and citations. But do please realise that its so-called "productivity boom" is likely to be largely caused by a better capturing of Australia's publications in the Web of Science, a situation it shares with virtually every other country in the world. As always, results of a bibliometrics analysis depend hugely on the database and on the timing of data collection. This is especially true for databases such as the Web of Science that have substantially expanded their coverage in order to catch up with new competitor products emerging after 2004 (Scopus, Google Scholar and Microsoft Academic).

Are research incentives driving behaviour?

Why would Australia display this particular pattern? Although it is difficult to find any conclusive evidence, my 2005 paper offered a range of conjectures. For many I provided a comparison with the UK and the Netherlands, the latter being similar in terms of population size. My money is on #6 as the most important reason.

1. High student/staff ratios: Government funding for universities has declined steadily in Australia; Australian universities generate a large part of their income from (international) student fees. This means they spend a lot of time and resources on recruitment and that student/staff ratios are generally very high. This is especially true in Business & Economics as this is the most popular choice for international students. At the University of Melbourne, student/staff ratios in this field were nearly five times as high as in Medicine and more than twice as high as in Engineering, leaving academics in Business & Economics less time to do high-impact research. My teaching load in Australia was certainly higher than it ever was in the UK and the Netherlands and my classes much bigger.
2. Less flexible salary structures: Australian universities are at a disadvantage to US universities in terms of the salaries they can offer. US universities can generally offer substantially higher salaries in order to attract top researchers from all over the world. However, this does not explain why Australia fares worse than for instance the UK and the Netherlands, neither of which are known for their generous academic salaries. Moreover, in the last decade Australian salaries have become more attractive in comparison to other countries and salary structures more flexible.
3. Research tied to Australian context: Australian research might be tied to the Australian context and hence be less interesting to international researchers. Articles published in top journals are indeed less likely to have an Australian focus. However, this does not explain why articles that are published are cited less. Of course this might indicate that North American researchers, who account for the bulk of citations in the field, are biased against Australian researchers. However, this does not really explain why Australia fares so much worse than for instance the UK and the Netherlands.
4. Business schools have a shorter history: Business education in Australia has a shorter history than in North America and the UK. However, while business schools might generally have been established much later in Australia than in the other countries, faculties of Economics & Commerce have a far longer history. And again this does not explain why Dutch academics fare better.
5. Australian research might simply reflect local managerial practice: Local Australian research might simply reflect local managerial practice. It is generally recognised in Australia that Australian managers have never been at the forefront of innovations in management. If Australian management lags behind its international counterparts, it might be more difficult for Australian academics to do innovative research to produce high quality articles.
6. Australian research evaluation rewards quantity over quality: In contrast to for instance the UK, the Netherlands and New Zealand, the Australian academic climate seems to reward quantity over quality. Yearly governmental research evaluations require the submission of all publications rather than just a sub-set. A journal article receives 1 point regardless of whether it is published in the field’s top journal or a third-tier journal. It is unclear why this policy would influence Economics & Business more so than other disciplines, although the discipline’s focus on instrumental rationality, efficiency & money might lead academics in this discipline to respond more strongly to incentives than e.g. Egyptologists. In addition, the pressured existence of academics in Economics & Business might lead them to go for the alternative that appears to maximise return on time invested in terms of publication points and promotion: publication in journals that are ISI listed, but have relatively low quality standards and high acceptance rates. Especially since the second half of the 1990s, Australia seems to lag behind the UK and the Netherlands in terms of publications in the top quartile of ISI rankings. The start of this period coincides with the introduction of quantity-based research evaluation in Australia.
7. Lack of research funding in Business & Management: I would normally expect competitively funded research to provide higher impact research output. A comparison of Australian Research Council Discovery grants for 2002–2005 showed that whereas funding in Economics was commensurate with its proportion of academic staff, Business & Management only accounted for 2.1% of the grants even though 12.6% of academic staff in Australian universities was employed in this field. In comparison, Engineering & Technology received 14.5% of the grants awarded for 6.9% of staff.

Conclusion

Australia’s lack of research impact – measured as citations per paper – in Economics & Business appears to be a persistent phenomenon. Unfortunately, many of the potential explanations refer to circumstances that are difficult to change. However, a change in research evaluation to a system that rewards quality as well as quantity might resolve some of these problems and can be implemented at the level of individual universities. In 2003 Linda Butler contrasted the differential research policies of the University of Queensland (focusing on quality) and the University of Western Australia (focusing on quantity). In 2016 UQ was the highest ranked Australian university in terms of citations per paper and UWA was one of the three lowest ranked. Food for thought…

References

• Butler, Linda (2003). Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts. Research Policy, 32(1): 143-155.

• Harzing, A.W. (2005) Australian research output in Economics & Business: High volume, low impact?, Australian Journal of Management, 30(2): 183-200. Available online... - Publisher’s version

Related YouTube video

Related blogposts

• To rank or not to rank
• Transcending the (non)sense of academic rankings
• Citation analysis for the Social Sciences: metrics and data-sources
• Health warning: Might contain multiple personalities
• Benchmarking research performance
• Why metrics can (and should?) be used in the Social Sciences

How to cite this blog?

A free pre-publication version of this paper is hyperlinked. If you’d like to have an official reprint, just drop me an email. If you refer to my work I would prefer you to cite the original paper. However, if you want to refer to the new information in this blog, please do so as follows:

• Harzing, A.W. (2017) Australian research output in Economics & Business: quantity over quality?, Harzing.com blog, http://www.harzing.com/blog/2017/05/australian-research-output-in-economics-business-quantity-over-quality.