9.1.2 Non-ISI listed publication can be high-quality publications
There is also a frequent assumption amongst research administrators that ISI listing is a stamp of quality and that hence one should ignore non-ISI listed publications and citations. There are two problems with this assumption. First, ISI has a bias towards Science, English-language and North American journals. Second, ISI completely ignores a vast majority of publications in the Social Sciences and Humanities.
ISI journal listing
ISI's listing of journals is much more comprehensive in the Sciences than in the Social Sciences and Humanities. Butler (2006) analyzed the distribution of publication output by field for Australian universities between 1999 and 2001. She found that whereas for the Chemical, Biological, Physical and Medical/Health sciences between 69.3% and 84.6% of the publications were published in ISI listed journals, this was the case for only 4.4%-18.7% of the publications in the Social Sciences such as Management, History Education and Arts.
As is documented in detail in Chapter 15, many high-quality journals in my field (Economics & Business) are not ISI listed. Only 30%-40% of the journals in Accounting, Marketing and General Management & Strategy listed on my Journal Quality List (already a pretty selective list) are ISI listed. There is no doubt that on average journals that are ISI listed are perceived to be of higher quality. In the analysis in Chapter 15, journals that are ISI-indexed in general had a significantly higher h-index (23.5 versus 11.5; t = 15.002, p < 0.001). However, there are a very substantial number of non-ISI indexed journals that have a higher than average h-index.
Very limited coverage of non-journal publications
Second, as has been mentioned many times in this book, even in the Cited Reference search ISI only includes citations in ISI listed journals and ignores any non-ISI publications for which you are not the first author. In the General Search function it completely ignores any publications that are not in ISI-listed journals.
As a result a vast majority of publications and citations in the Social Sciences & Humanities, as well as in Engineering & Computer Science, are ignored. In the Social Sciences and Humanities this is mainly caused by a completely neglect of books, book chapters, publications in languages other than English, and publications in non-ISI listed journals. In Engineering and Computer Science, this is mostly caused by a neglect of conference proceedings.
ISI has recently introduced conference proceedings in their database. However, it does not provide any details of which conferences are covered beyond listing some disciplines that are covered. I was unable to find any of my own publications in conference proceedings.
As a result ISI very seriously underestimates both the number of publications and the number of citations that academics in the Social Sciences & Humanities and in Engineering & Computer Science receive. A detailed analysis can be found in Chapter 16. However, busy Deans and tenure/promotion panel members might just want to read the bullet point conclusion, reproduced below.
- Academics in the Sciences out-perform academics in the Social Sciences and Humanities by a factor of 17.5 if one considers only the traditional performance indicator: ISI General Search citations.
- Academics in the Social Sciences and Humanities out-perform academics in the Sciences when using a more comprehensive data-source (Google Scholar) and correcting for career stage and the number of co-authors.