Proof over promise: a more inclusive ranking of academics

Shows how using a different data source and different metrics completely changes the Dutch Economics top-40

Anne-Wil Harzing - Thu 7 Jul 2016 09:01 (updated Fri 3 Jun 2022 08:22)

ranking

[This post was also published at the LSE Impact Blog as Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach].

Economists love to rank. Even a casual five-minute literature search reveals literally hundreds of publications on rankings of academic productivity and impact. Dutch economists are no exception. In fact, they produced what, to our best knowledge, is the oldest ranking in the field: a nation-wide ranking of Economists (the Economics top-40) that has entered its fourth decade. However, this ranking is based on publication volume in ISI-listed journals weighted by the journal’s impact factor or article influence score, rather than on the actual citation impact of the publications in question.

In this blog we will show that an impact driven approach using Google Scholar creates a more democratic and inclusive assessment. Proof over promise is a bibliometric paper dealing with rankings of academics, from a project I conducted with Wilfred Mijnhardt

Harzing, A.W.; Mijnhardt, W. (2015) Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business, Scientometrics, 102(1): 727-749. Available online... - Publisher's version (read for free)

Proof (impact) over Promise (journals)

On average, publications in high-impact journals by definition get cited more frequently than publications in low-impact journals as the journal impact factor is based on average citations. This still is the fundamental logic behind the creation and maintenance of Journal ranking list by business schools and associations such as the British Association of Business Schools. We call this principle “promise”, i.e. publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. However, not all individual papers published in these high-impact journals will fulfil this promise. We therefore set out to create a ranking based on “proof”, i.e. rather than looking at the promised number of citations implied by the journal impact factor or article influence score, we look at actual citations to an author’s work.

In our new ranking we used the hIa index, which is calculated by dividing the individual h-index (an h-index corrected for the number of co-authors) by the number of years an academic has been active, i.e. the number of years that have lapsed since their first publication. The metric thus represents the average number of single-author-equivalent “impactful” articles that an academic has published per year and hence permits an intuitive interpretation. As such, this metric provides a more reliable comparison between academics in different disciplines and at different career stages than the h-index.

A second aspect of our focus on proof over promise is the use of Google Scholar rather than ISI as a data source. Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research is published. If a particular research output is highly cited, it clearly influences the field and that should be more important than the journal in which it is published.

Resulting ranking for Economics & Business in the Netherlands

The resulting hIa-based ranking of academics in Economics & Business in the Netherlands is substantially different from the original ranking based on publications and is more inclusive in terms of sub-disciplines, age and affiliation.

Reduction of dominance of Economics over Business: Whereas in the original publication-based Economics top-40 more than two thirds of the listed academics works in Economics or Econometrics, this proportion is reduced to 43% in the hIa ranking. The change in disciplinary composition is particularly striking in the top-20. In the publication-based Economics top-40, three quarters of the academics in the top-20 were economists, in the PoP hIa top-40 this is reduced to just over a third.
Reduction of dominance of older academics: At 47, the average age in the hIa top-40 is lower than in the original ranking (50), but most noticeable is the addition an additional five academics under 40 and another five aged between 40 and 45. In fact, all but three of the newly listed academics in this top-40 are aged 45 or under.
Reduction of dominance of a small number of institutions: In the original top-40 academics affiliated with Erasmus and Tilburg made up just over or just under half of the list, in the hIa top-40, this is reduced to a third. Apart from the University of Utrecht, every university is represented in the hIa top-40. Maastricht, Eindhoven and the UvA (University of Amsterdam) in particular do much better in citation-based rankings than in the original publication-based top-40.

ranking

A more inclusive ranking?

We argue that there are four reasons why our “proof over promise” approach is more “democratic” and inclusive than the original Economics top-40.

Removal of disciplinary bias against Business: First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listed journals, we remove the disciplinary bias against Management, Marketing and Accounting & Finance, disciplines in which a smaller proportion of high-quality journals is ISI-listed than in Economics and Econometrics (Harzing & van der Wal, 2009).
Verdict is based on the entire academic community: Second, citation-based performance metrics can be argued to be more democratic as their “verdict” is based on the reception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal is dependent on only a handful of gatekeepers (the editor and reviewers).
Transparent and easy to replicate: Third, our ranking was created with the use a free software program (Publish or Perish) and a publicly available database (Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-based databases. This also means that any academic can look up their own citation record and easily find out where they “score” in the current ranking
More dynamic with better chances for younger academics: Fourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings in terms of changes over the years. Younger academics can more easily enter into the hIa ranking if they perform well relative to their career length in terms of single-author equivalent impactful papers.

Further references

Harzing, A.W.; Alakangas, S.; Adams, D. (2014) hIa: An individual annual h-index to accommodate disciplinary and career length differences, Scientometrics, 99(3): 811-821. Available online... - Publisher’s version
Harzing, A.W.; Wal, R. van der (2009) A Google Scholar h-index for journals: An alternative metric to measure journal impact in Economics & Business?, Journal of the American Society for Information Science and Technology, 60(1): 41-46. Available online... - Publisher’s version