8.4.1 Google Scholar comprehensively evaluates the impact of books

In the Social Sciences and Humanities book awards are a very important signal of academic excellence. In the Humanities, it can even be considered to be one of the most important indicators as books are normally seen as the most prestigious research outputs.

Several academics have informed to me that they have used Publish or Perish to propose one of their colleagues or students for a book award. Being able to show impact is especially important for relatively unknown authors. Hence, having a good understanding of the impact of research books is very important for award committees. Publish or Perish can be used to easily get a comprehensive account of citation data for books as the two examples below show.

The impact of books across disciplines and data sources

In the detailed comparisons of data sources and metrics across disciplines that is presented in Chapter 16, I noticed that all academics outside the Sciences had published at least one research book. The table below shows the citation impact of these books according to ISI and Google Scholar.

Discipline	ISI citations (cited reference search)	GS citations	% Increase GS citations
Business Academic	66	200	203%
Computer Scientist	None*/538	2599	383%
Education academic	132	797	504%
Linguist	78	518	564%
Political scientist	62	215	247%
Media studies academic	110	538	389%
Total	986	4858	394%

* The academic in question is 2nd author of the book, so the book does not appear when searching for his name in ISI.

Please note that the number of citations for ISI was acquired using the ISI Cited reference functions, as books do not show up at all in ISI's general search function (see Chapter 1 and Chapters 13 and 16 for further details about these differences). The many stray citations for these books (several dozens for the most cited book) were manually added to the counts.

It is abundantly clear that by only looking at ISI citations, I would seriously underestimate the impact of these research books. Even for the Business academic Google Scholar find three times as many citations, whilst for the Linguist Google Scholar finds nearly seven times as many citations. Overall, Google Scholar reports nearly five times as many citations as ISI for these research books.

Terry book award winner do have impact!

Our second illustration involves a citation analysis for books winning the Terry book award between 1992-2001. I have chosen this award for two reasons. First, the Terry Book award is a prestigious award given yearly by the Academy of Management, the most important professional organization in the field of management.

Second, Pfeffer and Fong (2002) strongly criticized business school research for its lack of impact. One of their arguments was based on an analysis of the impact of books winning the Terry Book award. Pfeffer and Fong (2002) claim that even these, supposedly highly influential, books only had an average of 6.80 citations per year.

Walsh, Tushman, Kimberly, Starbuck & Ashford (2007) subsequently use these data to conclude that this shows that our best books are not particularly well read even by our scholarly peers. I found this conclusion rather surprising and wondered whether the same conclusion would be drawn if I used a more inclusive source of citation impact, i.e. Google Scholar instead of ISI's Web of Science.

Inaccurate citation analysis creates myths

In order to assess this, I first repeated the analysis conducted by Pfeffer and Fong (2002). This analysis was conducted in September 2007. Our results show that the Terry Book Award winners on average received 346 citations in the Web of Science for an average of approximately 34 citations a year (calculated by dividing citations by the number of years since publication), not exactly a performance which I would consider to show low impact.

Hence our conclusion strongly contradicts that of Pfeffer and Fong. It is unclear why our analysis resulted in so many more citations. One reason might be the fact that our analysis was conducted five years later than theirs and hence the books had had more time to gather citations. This shows that the impact of books might take some time to effectuate and hence JIF-like indices that use two year time-spans would not be very useful for books or for research fields that take a long time for work to penetrate.

Another reason might be that Pfeffer and Fong did not systematically include misspellings or appearances with different author initials (i.e. the many stray citations in ISI). A final reason might be that in the case of the Handbook of Organization Studies they did not include citations to individual chapters.

Whatever the reason, it shows that one should be very careful before drawing rather far-reaching conclusions. Whilst I would not necessarily negate Pfeffer and Fongs general conclusion that business school do not have much impact on practice, their conclusion that even our best books do not have an academic impact or Walshs even stronger conclusion that “our best books are not particularly read by business people or by their scholarly peers” (Walsh et al., 2007: 129) does not seem to be supported by the data.

This note of caution is all the more important since myths are easily created by subsequent citations that seem to endorse the message, making it unassailable (see also Harzing, 2002). The Pfeffer & Fong article had already gathered 57 ISI and 169 GS citations in September 2007, when this analysis was initially conducted. Today (July 2010) it has no less than 165 ISI citations and a staggering 576 Google Scholar citations.

GS reports 2.5 times as many citations as ISI's Web of Science

However, even more remarkable than the difference between our Web of Science search and that of Pfeffer & Fong, is the difference in impact when using Google Scholar as a base for citations. On average GS reports nearly two-and-a-half times as many citations as the ISI WoS, for an average of 833 citations per book and 81 citations per book per year. Both measures show that these books have (had) a very considerable impact on the field.

The differences are particularly large for two books in the area of Strategic Management (those authored by Haspeslagh & Mintzberg), reflecting our observation in Chapter 15 that Strategy journals are not particularly well covered in ISI. In the case of Philip Haspeslagh, the fact that this academic is not working at a North American university might also have led to a modest impact in ISI listed journals, given the focus of ISI on North American journals.

Year	First Author	Title	ISI cites	GS cites	% GS incr.	ISI/year	GS/year
1992	Stopford, J.M.	Rival States, Rival Firms: Competition for World Market Shares	166	266	60%	11	18
1993	Haspeslagh, P.C.	Managing acquisitions: creating value through corporate renewal	125	513	310%	9	37
1994	Cox, T.	Cultural Diversity in Organizations: Theory, Research, and Practice	298	587	97%	23	45
1995	Mintzberg, H.	The Rise & Fall of Strategic Planning: Reconceiving Roles for Planning, Plans, Planners	480	1614	236%	40	124
1996	Rousseau, D.M.	Psychological Contracts in Org.: understanding written & unwritten agreements	406	951	134%	37	86
1997	Clegg, S.R.	The Handbook of Organization Studies	1315	2667	103%	132	267
1998	Nohria, N.	Differentiated Network: Organizing Multinational Corporations for Value Creation	101	270	167%	11	30
1999	Brown, S.L.	Competing on the edge: strategy as structured chaos	221	499	126%	28	62
2000	Aldrich, H.	Organizations Evolving	316	917	190%	45	131
2001	Thomas, D.A.	Breaking Through: The Making of Minority Executives in Corporate America	35	47	34%	6	8
Average			346	833	141%	34	81

An update of the Google Scholar citations for these books was done in July 2010 (i.e. nearly three years after the initial data collection), using the Publish or Perish general search function (see screenshot below). This showed that on average these books had gathered another 240 Google Scholar citations per year and the average total number of citations for the books was around 1660 citations, hardly a sign of a lack of impact.