15.3.2 Overall comparison w/h psychology journals and major outliers

Figure 2 (see the end of this chapter) shows a scatterplot of the average ISI JIF and Google Scholar h-index with the exclusion of Psychology journals and the three outliers discussed above: ACM Computing Surveys, Journal of Finance and the American Economic Review.

In this figure the most striking outlier is SIAM Review, with several Sociology/Geography journals also showing a high JIF in comparison to their h-index. Other less striking outliers are also visible in Figure 2, but these will be discussed in the review of the sub-disciplines below. SIAM Review had an average JIF of 2.75 between 2001 and 2003 and a JIF of 2.67 in 2006. However, in 2004 and 2005 its JIF was 6.12 and 7.21 respectively causing a very high average JIF between 2003 and 2006.

Reviewing the JIF for 2004 and 2005 in detail showed that the very high impact factor was nearly entirely caused by the very large number of citations to one particular journal article published in 2003 (The structure and function of complex networks by MEJ Newman). In October 2007 this particular article had been cited 998 times, twelve times more than the next highest cited article published in 2003. In fact, in October 2007 the Newman paper alone makes up for 80% of the citations to SIAM review in 2003; the other twenty papers published in 2003 together have only 249 citations. This example clearly shows the danger of relying on mean-value metrics, which can be heavily influenced by individual outliers.

The Sociology/Geography outliers are caused by a less extreme occurrence of the same problem, i.e. very concentrated citation scores. For the Annual Review of Sociology for instance, the top 3 (out of 104) papers make up nearly one third of the total number of citations. Hence whilst its JIF may be reasonably high, its h-index is modest, as citations taper off quickly after the first highly-cited papers.