8.4.1 Google Scholar comprehensively evaluates the impact of books
In the Social Sciences and Humanities book awards are a very important signal of academic excellence. In the Humanities, it can even be considered to be one of the most important indicators as books are normally seen as the most prestigious research outputs.
Several academics have informed to me that they have used Publish or Perish to propose one of their colleagues or students for a book award. Being able to show impact is especially important for relatively unknown authors. Hence, having a good understanding of the impact of research books is very important for award committees. Publish or Perish can be used to easily get a comprehensive account of citation data for books as the two examples below show.
The impact of books across disciplines and data sources
In the detailed comparisons of data sources and metrics across disciplines that is presented in Chapter 16, I noticed that all academics outside the Sciences had published at least one research book. The table below shows the citation impact of these books according to ISI and Google Scholar.
|Discipline||ISI citations (cited reference search)||GS citations||% Increase GS citations|
|Media studies academic||110||538||389%|
* The academic in question is 2nd author of the book, so the book does not appear when searching for his name in ISI.
Please note that the number of citations for ISI was acquired using the ISI Cited reference functions, as books do not show up at all in ISI's general search function (see Chapter 1 and Chapters 13 and 16 for further details about these differences). The many stray citations for these books (several dozens for the most cited book) were manually added to the counts.
It is abundantly clear that by only looking at ISI citations, I would seriously underestimate the impact of these research books. Even for the Business academic Google Scholar find three times as many citations, whilst for the Linguist Google Scholar finds nearly seven times as many citations. Overall, Google Scholar reports nearly five times as many citations as ISI for these research books.
Terry book award winner do have impact!
Our second illustration involves a citation analysis for books winning the Terry book award between 1992-2001. I have chosen this award for two reasons. First, the Terry Book award is a prestigious award given yearly by the Academy of Management, the most important professional organization in the field of management.
Second, Pfeffer and Fong (2002) strongly criticized business school research for its lack of impact. One of their arguments was based on an analysis of the impact of books winning the Terry Book award. Pfeffer and Fong (2002) claim that even these, supposedly highly influential, books only had an average of 6.80 citations per year.
Walsh, Tushman, Kimberly, Starbuck & Ashford (2007) subsequently use these data to conclude that this shows that our best books are not particularly well read even by our scholarly peers. I found this conclusion rather surprising and wondered whether the same conclusion would be drawn if I used a more inclusive source of citation impact, i.e. Google Scholar instead of ISI's Web of Science.
Inaccurate citation analysis creates myths
In order to assess this, I first repeated the analysis conducted by Pfeffer and Fong (2002). This analysis was conducted in September 2007. Our results show that the Terry Book Award winners on average received 346 citations in the Web of Science for an average of approximately 34 citations a year (calculated by dividing citations by the number of years since publication), not exactly a performance which I would consider to show low impact.
Hence our conclusion strongly contradicts that of Pfeffer and Fong. It is unclear why our analysis resulted in so many more citations. One reason might be the fact that our analysis was conducted five years later than theirs and hence the books had had more time to gather citations. This shows that the impact of books might take some time to effectuate and hence JIF-like indices that use two year time-spans would not be very useful for books or for research fields that take a long time for work to penetrate.
Another reason might be that Pfeffer and Fong did not systematically include misspellings or appearances with different author initials (i.e. the many stray citations in ISI). A final reason might be that in the case of the Handbook of Organization Studies they did not include citations to individual chapters.
Whatever the reason, it shows that one should be very careful before drawing rather far-reaching conclusions. Whilst I would not necessarily negate Pfeffer and Fongs general conclusion that business school do not have much impact on practice, their conclusion that even our best books do not have an academic impact or Walshs even stronger conclusion that “our best books are not particularly read by business people or by their scholarly peers” (Walsh et al., 2007: 129) does not seem to be supported by the data.
This note of caution is all the more important since myths are easily created by subsequent citations that seem to endorse the message, making it unassailable (see also Harzing, 2002). The Pfeffer & Fong article had already gathered 57 ISI and 169 GS citations in September 2007, when this analysis was initially conducted. Today (July 2010) it has no less than 165 ISI citations and a staggering 576 Google Scholar citations.
GS reports 2.5 times as many citations as ISI's Web of Science
However, even more remarkable than the difference between our Web of Science search and that of Pfeffer & Fong, is the difference in impact when using Google Scholar as a base for citations. On average GS reports nearly two-and-a-half times as many citations as the ISI WoS, for an average of 833 citations per book and 81 citations per book per year. Both measures show that these books have (had) a very considerable impact on the field.
The differences are particularly large for two books in the area of Strategic Management (those authored by Haspeslagh & Mintzberg), reflecting our observation in Chapter 15 that Strategy journals are not particularly well covered in ISI. In the case of Philip Haspeslagh, the fact that this academic is not working at a North American university might also have led to a modest impact in ISI listed journals, given the focus of ISI on North American journals.
|Year||First Author||Title||ISI cites||GS cites||% GS incr.||ISI/year||GS/year|
|1992||Stopford, J.M.||Rival States, Rival Firms: Competition for World Market Shares||166||266||60%||11||18|
|1993||Haspeslagh, P.C.||Managing acquisitions: creating value through corporate renewal||125||513||310%||9||37|
|1994||Cox, T.||Cultural Diversity in Organizations: Theory, Research, and Practice||298||587||97%||23||45|
|1995||Mintzberg, H.||The Rise & Fall of Strategic Planning: Reconceiving Roles for Planning, Plans, Planners||480||1614||236%||40||124|
|1996||Rousseau, D.M.||Psychological Contracts in Org.: understanding written & unwritten agreements||406||951||134%||37||86|
|1997||Clegg, S.R.||The Handbook of Organization Studies||1315||2667||103%||132||267|
|1998||Nohria, N.||Differentiated Network: Organizing Multinational Corporations for Value Creation||101||270||167%||11||30|
|1999||Brown, S.L.||Competing on the edge: strategy as structured chaos||221||499||126%||28||62|
|2000||Aldrich, H.||Organizations Evolving||316||917||190%||45||131|
|2001||Thomas, D.A.||Breaking Through: The Making of Minority Executives in Corporate America||35||47||34%||6||8|
An update of the Google Scholar citations for these books was done in July 2010 (i.e. nearly three years after the initial data collection), using the Publish or Perish general search function (see screenshot below). This showed that on average these books had gathered another 240 Google Scholar citations per year and the average total number of citations for the books was around 1660 citations, hardly a sign of a lack of impact.