Author evaluation: Tips for Deans

Publish or Perish tutorial

Anne-Wil Harzing - Sat 6 Feb 2016 16:10 (updated Thu 2 Jun 2022 11:01)

Publish or Perish was initially designed to help academics to present their case for research impact to its best advantage. It tends to be used mostly by individual academics that have full knowledge of their own publication records and thus can make sensible conclusions about the resulting data.

Inevitably, however, it also became popular amongst Deans, academic administrators and chairs of tenure/promotion committees. I am not very comfortable with the mechanistic type of evaluation that might be promoted by an exclusive focus on citation analysis. However, I do realize that many tenure or promotion committees use this kind of information. So what advice can I give on citation analysis to administrators who want to maintain rigorous standards, but also want to be fair and equitable?

Give up your reservations about Google Scholar

First, especially if you work in the Social Sciences and Humanities, or in Engineering and Computer Science, give up your reservations about Google Scholar. You can always ask applicants to provide ISI or Scopus data as well, but do take Google Scholar data seriously. These days, many universities allow or even recommend the use of Google Scholar and Publish or Perish. Institutions such World Bank and Microsoft’s research laboratory use it to evaluate their broader impact beyond ISI listed journals. Many US government departments use it to evaluate the impact of particular research projects.

Yes, some of Google Scholar's results are nonsensical. Yes, occasionally Google Scholar will double-count citations. Yes, some of the citations are not as scholarly as one would want. However, these minor errors are not worth worrying too much about when comparing them with the systematic and very large underestimation of citations in ISI or even Scopus for many disciplines.

Give up your fixation with self-citations

Second, give up your fixation with self-citations. Yes, occasionally academics abuse the system and systematically cite their own work. However, there are safeguards against this system as journal editors and reviewers will not like gratuitous citations. Moreover, if academics are so unethical, they probably have bigger problems than their inflated citation record. More importantly, for the majority of academics self-citations do not distort their citation records in any significant way. The hassle and possible inconsistency in excluding self-citations is not worth the small possible gain in accuracy.

Don't apply citation analysis for junior academics

Third, be very hesitant in applying citation analysis for junior academics, especially in the Social Sciences and Humanities. It can easily take 5-10 years after an academics first publication for a significant number of citations to flow in. Hence, if an early career academic shows a large number of citations, make sure you promote them and keep them happy (assuming other aspects of their performance are also at least satisfactory). However, if academics going up for tenure have very few citations, don’t hold it against them.

Realize citations vary dramatically between disciplines

Fourth, realize that citations can vary dramatically between or even within disciplines. Never compare citation (or publication) records across disciplines. If for some reason you have to do so, use Google Scholar, not ISI or Scopus and preferably use a discipline corrected metric, such as the hI,annual (see reference). Be very hesitant to prescribe norm scores for the number of citations to be accumulated before someone is considered for tenure or promotion to a certain level.

If you feel you do need to prescribe norm scores, make sure they are realistic and reflect actual performance of those at similar levels. Too many senior academics seriously overestimate the number of citations they had themselves when being promoted! I have had my Department Head describe my citation levels as modest in a reference letter; they were more than twice as high as his own when he was promoted at a top US university. My citation levels were also higher than those of all but one of the professors in our department, some of whom had been in post for a decade or more.

Consult an expert

Fifth, if you have any doubts about what you are doing, consult an expert. Academics and administrators at this level are far too busy to try to understand the minutia of citation analysis. Get a proper bibliometric expert involved! You might have access to a librarian with good skills in this respect or you might have academics in your staff who do bibliometric research. If you do not, you can read Chapter 9 of the Publish or Perish Book, which provides more detail on all of the issues above. If you read it closely, you’ll probably know more about citation analysis than 99.9% of the academics. It is freely accessible online: http://www.harzing.com/popbook/ch9.htm

References

• Harzing, A.W.; Alakangas, S.; Adams, D. (2014) hIa: An individual annual h-index to accommodate disciplinary and career length differences, Scientometrics, 99(3): 811-821. Available online... - Publisher’s version

Support Publish or Perish

The development of the Publish or Perish software is a volunteering effort that has been ongoing since 2006. Download and use of Publish or Perish is and will remain free (gratis), but your support toward the costs of hosting, bandwidth, and software development are appreciated. Your support helps further development of Publish or Perish for new data sources and additional features.

Feedback

PS: If you are using Publish or Perish on a regular basis, please take 5 minutes to provide me with some feedback.