1.4.1 Thomson ISI journal impact factor
One of the earliest citation metrics that became widely used is Thomson ISI's (now Thomson Reuters) Journal Impact Factor (or JIF). A simple short-hand explanation of the JIF is that it reflects the average number of citations that can be expected for an article in the journal in question within 1-2 years after publication.
A more formal definition is the number of citations in a particular year (e.g. 2009) to all items published in the journal in question in the two years before (e.g. 2007-2008) divided by the number of source items. Source items are regular articles, i.e. excluding letters, book reviews and editorials, conference abstracts. However, citations to non-source items are included in the numerator.
Thomson ISI publishes Journal Impact Factors in June every year for the previous year, i.e. JIFs for 2009 come out in June 2010. As typical citation patterns vary by discipline JIFs vary enormously by discipline. In the Social Sciences and Humanities impact factors above 2.0 are rare, in the Sciences they are quite common. This means that the average article in the Social Sciences and Humanities is cited less than once a year.
There are several commonly mentioned problems (see e.g. Seglen, 1997; Cameron, 2005 for a summary) with the use of the ISI Journal Impact Factor, the most important of which are the use of a 2-year citation window and technical issues related to the calculation of the JIF.
2-year window ignores field differences in citation patterns
When JIFs were introduced by Garfield in the 1960s, his focus was on biochemistry and molecular biology, disciplines that are characterized by a high number of citations and short publication lags (Cameron, 2005). Hence the use of a 2-year citation window might have been justified. However, this is not true for most other disciplines where knowledge takes much longer to be disseminated. Although Thomson ISI recently introduced a 5-year impact factor, the 2-year impact factor is still the most commonly used.
As Leydesdorff (2008) shows, impact factors can differ with an order of magnitude when comparing across disciplines such as Mathematics and Genetics. McGarty (2000) discusses the problems associated with the 2-year JIF for the Social Sciences in some detail. He shows that the publication lags for two important Psychology journals are such that for a typical paper published in these journals, two thirds of the literature that could theoretically be included in the JIF (i.e. papers published in the two years preceding publication of the referencing paper) was yet to be published at the time of submission.
A perusal of the last issue of 2007 of the Journal of International Business Studies shows that the problem is at least as severe in Business Studies. Even in this most optimistic case (i.e. the final issue of 2007) we find very few references to publications in 2005 and 2006 in the ten articles published in this issue. Out of the more than 700 references in this issue, only 20 referred to publications in 2005 and a mere 7 to publications in 2006 (i.e. less than 4% of the total number of citations).
One third of these citations were self-citations. This is not entirely surprising given that of the ten papers in this issue, six were submitted before 2005 (four in 2004, one in 2003, one in 2002). Of the remaining four, two were submitted in January and February 2005 and hence cannot realistically be expected to include references to 2005 papers. The final two papers were submitted in January and May 2006. The fact that we find any references to papers published in 2005 and 2006 in these articles is most likely due to these references being included in the review process. As McGarty (2000:14) aptly summarizes:
The two year impact factor clearly favors journals which publish work by authors who cite their own forthcoming work and who are geographically situated to make their work readily available in preprint form. The measure punishes journals which publish the work of authors who do not have membership of these invisible colleges and is virtually incapable of detecting genuine impact. It is not just a bad measure, it is an invitation to do bad science.
Technical and statistical problems in calculation of the JIF
In addition to the limitations associated with a 2-year window, there are several technical or statistical problems with the way the JIF is calculated. First, whilst the denominator in the JIF (the number of articles published) only includes normal articles (so called source items), the numerator includes citations to all publications in the journal in question, including editorials, letters, and book reviews (Cameron, 2005). This means that citations in these latter publications are basically “free” as the increase in the numerator is not matched by an increase in the denominator. As a result, journals with a lively letters to the editor/correspondence section (such as for instance Nature) will show inflated JIFs.
This problem is compounded by the fact that many journals have increased the proportion of non-source items over time. Gowrishankar & Divakar (1999) indicate the proportion of source items to non-source items in Nature declined from 3.5 to 1.6 between 1977 and 1997. This particular JIF feature also enables manipulation of the JIF by unscrupulous editors who can inflate their JIF by referring frequently to journal articles or even to other editorials in their editorials (Whitehouse, 2002). Bayliss, Gravenor & Kao (1999) argue that even a single research institute could increase the JIF of a journal that publishes few papers from 1 to 6 by asking each of its researchers to cite two papers in that journal.
The second calculation problem is statistical in nature: the JIF calculates the mean number of citations to an article in the journal in question. However, many authors have found that citation distributions are extremely skewed. Seglen (1997) for instance found the most cited 15% of papers to account for 50% of citations and the most cited 50% for 90% of the citations. Hence on average the most cited half of papers are cited nine times as much as the least cited half. Especially for journals publishing a relatively small number of papers, individual highly cited papers have a very strong influence on the mean JIF.