15.2 Methods

Since our aim was to cover a broader range of journals than in most previous studies I took the Harzings Journal Quality List (Harzing, 2007) as our basis. This list includes a collation of twenty different rankings of 838 journals in the broad area of Business and Economics.

The metrics used in this chapter were calculated using Publish or Perish. Searches were conducted in the first week of October 2007. Where relevant I searched for spelling variations of a journal (e.g. British vs. American spelling, the use of and vs. the use of &, spelling of composite words with or without a hyphen). Some journals also have abbreviated titles that are commonly used (e.g. all SIAM journals and many Psychology journals) and hence these were included as alternatives. If a title included very common words, e.g. Journal of Management, I conducted searches with the ISSN instead. Unfortunately, Google Scholar's results for ISSN searches seem to be rather erratic and hence this alternative was only used if the ISSN search provided a comprehensive result for the journal in question.

The results of all search queries were inspected for incomplete or inconsistent results. This process left us with only two dozen journals (out of 838) that had substantially incomplete coverage and for which metrics could not be calculated. Eight of these were research annuals in book format (the Elsevier Advances in … series and the Research in…. series). For other journals our visual inspection might have overlooked occasional missing articles, but this is unlikely to impact much on robust measures such as the h-index and g-index unless they happen to be highly cited. I have no reason to believe that this was the case. On the contrary, highly cited articles appear to be less likely to be missing from the Google Scholar database than lowly cited or uncited articles.

Our Google Scholar searches included citations to articles published between 2001-2005. This timeframe was chosen to be broadly comparable with a five-year average for the Journal Impact Factors of the last five available years (2003-2006). These impact factors refer to citations in articles published between 2001 and 2005. Ideally, I would have preferred to include the JIF for 2007, but that metric did not come out until half a year after the analysis conducted for this chapter. Moreover, given that Google Scholar displays some delay in its data processing for some journals, using the 2006 JIF is likely to give a dataset that is fairly comparable to Google Scholar in October 2007.

Supplementary analyses reported below with regard to the extent of concentration of citations within a particular journal were conducted in October 2007 with the general search function of ISI that allows the user to rank articles by citation.