An Australian "productivity boom"? ... or maybe just a database expansion?
Shows how difficult longitudinal bibliometric analyses are when database coverage changes
A "good news" story about the "productivity boom" of Australian researchers in the 12 July 2018 issue of University World News prompted me to update my blogpost Australian research output in Economics & Business: quantity over quality?. The University World News story is based on an analysis conducted by "academics at the Innovative Research Universities (IRU) group". In their report, they compared Web of Science publications for 2016 with publications for 2006 and concluded that Australia had experienced a very high level of increased productivity, with publication output more than doubling in a decade. This was true even though the number of academics in Australia had only increased by a third over the same time, thus leading to the claim of a "productivity boom".
Although the study does acknowledge that China's increase in productivity far outstrips that of Australia, its main conclusion appears to be that Australia has performed much better than the UK and the USA, which only showed increases of 49% and 30% respectively. A similarly postive story is told about citations. Whilst I do not doubt the publication and citation figures are correct, I think it is a mistake to attribute them solely to higher levels of productivity and impact, let alone draw the conclusion that these positive developments are to be attributed to Australia's research funding system.
Comparing Australia with other countries
In order to illustrate this point, I will compare Australia's changing performance more systematically with that of other countries. Doing a country-by-country comparison for individual years is very time-consuming in the Web of Science interface. To replicate the report's analysis, I therefore used Clarivate's Web of Science "Essential Science Indicators" which ranks countries (as well as universities and individuals) on the number of publications and citations, an approach I also followed for the original paper on which this blogpost is based. I compared the 30-odd "most-publishing countries", using my original 1994-2004 data on the one hand and the most recent data-set (i.e. 2008-2018) on the other hand. Although this comparison is not identical to a comparison between 2016 and 2006 only, the results will be quite similar as publication and citation trends tend to change only slowly. My comparison showed that Australia had indeed quite dramatically increased both its number of publications and its number of citations, but so had most other countries.
Australia had increased its number of publications by 147% (i.e. even more than the 112% that was reported for the 2006 to 2016 period); the UK (56%) and the USA (43%) indeed showed only modest increases, "beating" only Russia, the Ukraine and Japan. However, beyond China there were eight other countries that showed a level of increase in overall productivity that was higher than Australia's: Turkey, Brazil, South Korea, Singapore, India, Ireland, South Africa, Taiwan and Poland. Moreover, there was a wide range of Western countries showing levels of increased productivity that were very similar to Australia: Spain, Norway, Greece, Denmark, New Zealand, Belgium, Austria, Switzerland, Italy and the Netherlands. With regard to the increase in the number of citations, there were even more countries that outperformed Australia.
Database expansion: regional journals
However, much of the increase for all countries involved can likely be attributed to an expansion of the Web of Science database. Over the past decade the Web of Science has substantially increased its coverage of "regional" (regional being "WoS speak" for Asia-Pacific, European, Latin-American, and Middle-Eastern & African journals) journals, books and book chapters. In 2008 alone, the Web of Science added 700 new regional journals. No less than 52 of these journals were Australian, making Australia the country with the largest number of journals added to the Web of Science database. The only other countries that saw a significant number of their journals added were Germany (50), Poland (43) and Spain (43), all much larger countries in terms of their population.
In Economics and Business, the discipline studied in the paper this blogpost is based on, Australia's relative increase in publications was even less spectacular: there are no less than fourteen countries with a more subtantial increase and another half dozen with a very similar increase. The picture for citations is identical. This is all the more suprising as no less than a third of the journals added in Economics & Business were Australian journals. Thus one could argue that we could have expected a much more impressive increase in "productivity" for Australia.
No change in Australia's worlwide ranking
As a result, even though Australia dramatically increased its number of publications and citations covered in the Web of Science, there is very little movement in the relative ranking of countries, neither by number of publications nor by citations per paper. No less than 14 years after the original data collection, Australia still ranks #11 for the number of publications in all disciplines combined, and - just like in 2004 - it is narrowly beaten by Spain at #10. The expansion of coverage and developments in publishing behaviour in emerging economies, however, have catapulted China from rank #9 to rank #2. Moreover, India now outranks Australia (moving from #13 to #9) and South Korea and Brazil have likewise improved their ranking (from #16 and #19 to #12 and #13). South Korea might wel eclipse Australia soon, having only 4% fewer publications in the WoS. In terms of citations per paper, Australia's rank likewise has not moved an inch: it is still at #17. Virtually all developed Western countries, as well as Singapore, have higher impact scores than Australia. The only countries in this group that Australia leaves behind are Italy, New Zealand, Spain, Greece and Portugal.
For Economics & Business, Australia has slightly improved its rank in terms of publications as it is now ranked a solid #4 rather than a shared #4. However, this is mainly caused by Canada dropping to #5 as Germany, which was originally ranked ex-equo with Australia, has increased its publications more than Australia and is now ranked #3. In terms of citations per paper Australia's rank has also slightly improved, now standing at #18 instead of at #19. However, Australia still has one of the largest productivity/impact gaps, only China and Japan do worse in this respect. In terms of citations per paper, Australia was outranked not just by the USA, the UK, and Canada, but also by most European countries (only Italy, Portugal, Spain, and Greece do worse), Singapore, Israel and New Zealand. The latter improved its citations per paper ranking from #18 to #15 and thus increased its lead on its bigger neighbour.
Celebrate, but celebrate cautiously
By all means celebrate the fact that Australia continues continues to pull well above its population weight in terms of publications and citations. Also celebrate that on a per capita basis Australia has slightly improved its world ranking for both publications and citations. But do please realise that its so-called "productivity boom" is likely to be largely caused by a better capturing of Australia's publications in the Web of Science, a situation it shares with virtually every other country in the world. As always, results of a bibliometrics analysis depend hugely on the database and on the timing of data collection. This is especially true for databases such as the Web of Science that have substantially expanded their coverage in order to catch up with new competitor products emerging after 2004 (Scopus, Google Scholar and Microsoft Academic).
Related blogposts
- Australian research output in Economics & Business: quantity over quality?
- To rank or not to rank
- Transcending the (non)sense of academic rankings
- Citation analysis for the Social Sciences: metrics and data-sources
- Health warning: Might contain multiple personalities
- Benchmarking research performance
- Why metrics can (and should?) be used in the Social Sciences
Copyright © 2022 Anne-Wil Harzing. All rights reserved. Page last modified on Sun 24 Apr 2022 14:57
Anne-Wil Harzing is Emerita Professor of International Management at Middlesex University, London. She is a Fellow of the Academy of International Business, a select group of distinguished AIB members who are recognized for their outstanding contributions to the scholarly development of the field of international business. In addition to her academic duties, she also maintains the Journal Quality List and is the driving force behind the popular Publish or Perish software program.
