Citation analysis: Tips for Deans and other administrators

Provides recommendations for Deans and senior administrators for fair and equitable research evaluation

Anne-Wil Harzing - Mon 25 Sep 2017 14:58 (updated Thu 2 Jun 2022 14:35)

My software program - Publish or Perish - was initially designed to help academics to present their case for research impact to its best advantage. It tends to be used mostly by individual academics that have full knowledge of their own publication records and thus can make sensible conclusions about the resulting data.

Inevitably, however, it also became popular amongst Deans, academic administrators and chairs of tenure/promotion committees. I am not very comfortable with the mechanistic type of evaluation that might be promoted by an exclusive focus on citation analysis. However, I do realize that many tenure or promotion committees use this kind of information. So what advice can I give on citation analysis to administrators who want to maintain rigorous standards, but also want to be fair and equitable?

Give up your reservations about Google Scholar

First, especially if you work in the Social Sciences and Humanities, or in Engineering and Computer Science, give up your reservations about Google Scholar. You can always ask applicants to provide ISI or Scopus data as well, but do take Google Scholar data seriously. These days, many universities allow or even recommend the use of Google Scholar and Publish or Perish. Staff members in institutions such as the World Bank and Microsoft’s research laboratory use it to evaluate their broader impact beyond ISI listed journals. Many US government departments use it to evaluate the impact of particular research projects.

Yes, some of Google Scholar's results are nonsensical. Yes, occasionally Google Scholar will double-count citations. Yes, some of the citations are not as scholarly as one would want. However, in my opinion these minor errors are not worth worrying too much about when comparing them with the underestimation of citation counts in ISI or even Scopus for academics in the Social Sciences and Humanities or Engineering and Computer Science.

Give up your fixation with self-citations

Second, give up your fixation with self-citations. Yes, occasionally academics abuse the system and systematically cite their own work. However, there are safeguards against this system as journal editors and reviewers will not like gratuitous citations. Moreover, if academics are so unethical, they probably have bigger problems than their inflated citation record. More importantly, for the majority of academics self-citations do not distort their citation records in any significant way. The hassle and possible inconsistency in excluding self-citations is not worth the small possible gain in accuracy. The only exception I would make here is that some less scrupulous scholars have managed to inflate their citation records through excessive self-citation, coupled with publication in predatory open access journals.

junior

Don't apply citation analysis for junior academics

Third, be very hesitant in applying citation analysis for junior academics, especially in the Social Sciences and Humanities. It can easily take 5-10 years after an academics first publication for a significant number of citations to flow in. Hence, if an early career academic shows a large number of citations, make sure you promote them and keep them happy (assuming other aspects of their performance are also at least satisfactory). However, if academics going up for tenure have very few citations, don’t hold it against them.

Realize citations vary dramatically between disciplines

Fourth, realize that citations can vary dramatically between or even within disciplines. Never compare citation (or publication) records across disciplines. If for some reason you have to do so, use Google Scholar, not ISI or Scopus and preferably use a discipline corrected metric, such as the hI,annual (see reference). Be very hesitant to prescribe norm scores for the number of citations to be accumulated before someone is considered for tenure or promotion to a certain level.

If you feel you do need to prescribe norm scores, make sure they are realistic and reflect actual performance of those at similar levels. Too many senior academics seriously overestimate the number of citations they had themselves when being promoted! I have had my Department Head describe my citation levels as "developing" in a promotion reference letter; they were more than twice as high as his own when he was promoted at a top US university.

Consult an expert

Fifth, if you have any doubts about what you are doing, consult an expert. Academics and administrators at this level are far too busy to try to understand the minutia of citation analysis. Get a proper bibliometric expert involved! You might have access to a librarian with good skills in this respect or you might have academics in your staff who do bibliometric research. If you do not, you can read Chapter 9 of the Publish or Perish Book, which provides more detail on all of the issues above. If you read it closely, you’ll probably know more about citation analysis than 99.9% of the academics. It is freely accessible online.

References

Harzing, A.W.; Alakangas, S.; Adams, D. (2014) hIa: An individual annual h-index to accommodate disciplinary and career length differences, Scientometrics, vol. 99, no. 3, pp. 811-821. Available online... - Publisher's version (read for free)