Citation analysis across disciplines: The impact of different data sources and citation metrics

Documents how different data ources and different citations metrics might lead to very different conclusions in cross-disciplinary comparisons

Anne-Wil Harzing - Sat 6 Feb 2016 16:10 (updated Sat 15 Apr 2023 07:43)

Prof. Anne-Wil Harzing, University of Melbourne
Web: www.harzing.com
Email: anne@harzing.com

© Copyright 2010 Anne-Wil Harzing. All rights reserved.

First version, 1 June 2010.

A vastly expanded version of this basic premise, including data  for 146 academics, was published in Scientometrics.

• Harzing, A.W.; Alakangas, S. (2016) Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison, Scientometrics, vol. 106, no. 2, pp. 787-804. Available online... - Publisher's version - Presentation slides - Video presentation of this article.

Abstract

Citation analysis is an increasingly common way to evaluate research impact. However, there seems to be a general lack of understanding of how different data sources and citation metrics might impact on comparisons between disciplines. This paper analyses the citation records of ten full professors at the University of Melbourne (Australia) in a variety of disciplines to illustrate how different data sources and different citations metrics might lead to very different conclusions.

If one considers the traditional performance indicator (ISI General Search citations), academics in the Sciences out-perform academics in the Social Sciences and Humanities. When using a more comprehensive data-source and correcting for career stage and the number of co-authors, Academics in the Social Sciences and Humanities out-perform academics in the Sciences.

Keywords: bibliometrics, citation analysis, research impact, research quality, h-index

Introduction

Citation analysis is an increasingly common way to evaluate research impact. However, there seems to be a general lack of understanding of how different data sources and citation metrics might impact on comparisons between disciplines. This paper analyses the citation records of ten full professors at the University of Melbourne (Australia) in a variety of disciplines to illustrate how different data sources and different citations metrics might lead to very different conclusions.

The ten professors were chosen at random, but they were all established professors in their fields. In many cases they had high level positions such as Head of Department, Director of a major research centre, Associate Dean Research, Dean of a School or Faculty, or Deputy Vice Chancellor. One should therefore not necessarily consider their performance to reflect typical norms scores in their disciplines. There might be many excellent professors with lower research impact scores.

In addition, I do not claim that the academics used as an example in this white paper are fully representative of their disciplines. There are many performance differences even within the same discipline. However, the results presented in this paper are fairly typical of the results I have gathered in nearly four years of research in citation analysis. This includes anecdotal knowledge gathered in responding to the many requests for assistance in using Publish or Perish, a citation analysis program using Google Scholar data.

Data Source Comparisons:
Total Number of Citations across Disciplines

Three different data sources for citation analysis are investigated in this white paper:

Thomson Reuters Web of Science (generally known as ISI Web of Science or ISI). This is the traditional source of citation data, established by Eugene Garfield in the 1960s. Many universities still use this as their only source of citation data. It has complete coverage of citations in the more than 10,000 journals that are ISI listed, going back to 1900. It is generally updated once or twice a week. Although its worldwide coverage has been improving recently, it still has a North American bias in many disciplines. It charges commercial rates for access.

Scopus. Introduced by Elsevier in 2004, Scopus aims to be the most comprehensive Scientific, Medical, Technical and Social Science abstract and citation database containing all relevant literature, irrespective of medium or commercial model. It covers nearly 18,000 titles from more than 5,000 publishers. It also claims worldwide coverage; more than half of Scopus content is said to originate from Europe, Latin America and the Asia Pacific region. It is updated daily and charges commercial rates for access.

Google Scholar. Introduced by Google in 2004, Google Scholar has become a very popular alternative data source, not least through the fact that access is free and citation analysis programs such as Publish or Perish make bibliometric analysis easy. Some academics are sceptical about its wider coverage. However, studies (e.g. Vaughan and Shaw (2008) have found most of the citations to be scholarly. After a relatively slow start Google Scholar coverage is increasing, although Google still does not provide a list of its sources. Google Scholar is updated several times a week. For a more detailed analysis about Google Scholar as a source for citation analysis see Harzing & van der Wal (2008).

Table 1 reports the number of citations for our ten University of Melbourne professors. Definitions of data coverage of the citation data sources can be found underneath the table. As indicated above, most universities still use ISI as their primary or even only source of citation data. We will therefore first compare citation records for Scopus and Google Scholar with ISI citations records. Subsequently, we will compare the two different types of search functions in both ISI (General and Cited By search) and Scopus (General and More search).

Table 1: Number of citations for different disciplines using different data sources

ISI General search = citations IN ISI-listed journals TO publications ISI-listed journals.

ISI Cited by search = citations IN ISI-listed journals TO all publications (incl. non ISI-listed journals, books, conference papers, white papers, government reports). Please note that even in the “cited by” search function ISI ignores citations for second and further authors for non-ISI publications.

Scopus General search = citations IN Scopus-listed journals TO publications in Scopus-listed journals.

Scopus More = citations IN Scopus-listed journals TO all publications (incl. non-Scopus listed journals, books, conference proceedings, white papers, government reports). Unlike the ISI Cited By search, the Scopus More search is not additive, i.e. in order to establish an academic’s total Scopus citations, one needs to add up the results from the Scopus General search and the Scopus More search.

Google Scholar = citations IN all publications (incl. academic journals that are not ISI or Scopus listed, books, conference proceedings, white papers, government reports) TO all publications (incl. academic journals that are not ISI or Scopus listed, books, conference proceedings, white papers, government reports).

Scopus versus ISI

The difference in citation records between ISI versus Scopus varies hugely by discipline. For the academics working in the Sciences, Scopus generally finds fewer citations than ISI, with the exception of our Computer Scientist. For the academics working in the Social Sciences and Humanities Scopus generally finds more citations than ISI, with the exception of the Cinema Studies academic.

A. Sciences

As is readily apparent from Figure 1, the pattern of reduced citation scores for the Scientists is most pronounced for the Pharmacologist who sees his citations reduced by more than 50%. His most cited article has 919 citations in ISI, but only 248 in Scopus. The simple reason for this is that Scopus only includes citations from 1996 onwards. In fact, Scopus and ISI provide a virtually identical number of citations for this article from 1996 onwards. As this particular academic has been publishing for more than 40 years, his citation record in Scopus is very incomplete.

Figure 1: Number of citations for ISI and Scopus General Search: Science disciplines

The Cell Biologist, Mathematician and Physicist also experience drops of around 25%, even though they have only been publishing for around 25 years. Again, this is caused by the fact that Scopus does not include citations before 1996 and all of these academics published some articles before this date. Comparison of individual articles published after 1995 shows virtually identical citation records in ISI and Scopus. If anything, Scopus tends to show a marginally higher number of citations for these articles.

In contrast, our Computer Scientist sees his citations increase by 43% in Scopus. There are several reasons for this. First, although this academic has been publishing for 31 years, many of his most cited articles were published after 1995 and hence Scopus’ restricted data coverage is not a problem. Second, the Computer Scientists benefits from the broader data coverage of Scopus in his field. Whilst ISI lists only 62 articles for him, Scopus lists around 100 articles. This difference did not occur for the other academics working in the Sciences.

B. Social Sciences and Humanities

For four of our academics working in the Social Sciences and Humanities (Business, Education and Linguistics), Scopus finds more citations than ISI. As shown in Figure 2, this pattern is pronounced for the Business academic who sees her citations increase by 77%. As she only started publishing in 1995, the lack of citation coverage before 1996 is not a problem.

Moreover, Scopus lists an additional 10 articles for her in journals that are not ISI-listed, but are included in the Scopus database. This includes her most highly cited article. In addition, because Scopus has a wider journal coverage in Business than ISI, citations for all her articles tend to be at least 10%, but sometimes 50% higher than citations in ISI.

Figure 2: Number of citations for ISI and Scopus General Search: Social Science and Humanities disciplines

A similar pattern is found for the academic working in Education. He even sees his citations increase by 90%, largely because Scopus lists more of the journals he has published in, but also because Scopus citations to articles listed in both databases are 20-100% higher than ISI citations.

The Linguist and the Political Scientist only show a modest increase by 18-20% as for them better journal coverage in Scopus is counterbalanced by a reduction in pre-1996 citations. However, for the journal articles that are listed in both sources, Scopus generally provides 20-80% more citations than ISI. Hence journal coverage in four of the Social Sciences and Humanities fields seems much broader in Scopus.

The Cinema Studies academic is worst off when using the Scopus database as nearly all of her journal publications date from before 1996, and the journals she has published in do not have pre-1996 coverage in Scopus. However, the only article by this academic that is included in Scopus has more citations than in ISI.

C. Conclusion

Comparing ISI and Scopus as a source for citations provides mixed results. In general, Scopus provides a higher citation count than ISI, both in the Sciences and in the Social Sciences and Humanities. In the Sciences, this increase in only marginal (except for Computer Science), whilst in the Social Sciences and Humanities, this increase is substantial.

Scopus appears to have a much broader journal coverage for the Social Sciences and Humanities than ISI and hence provides a fairer comparison. Whilst in ISI academics working in the Sciences have on average 17.5 times as many citations as the academics working in the Social Sciences and Humanities, in Scopus this difference is reduced to 7.5 times.

However, for the time being Scopus is hindered by its lack of coverage before 1996. This means that for most established academics in the Sciences, Scopus will lead to lower lifetime citation counts than ISI. In the Social Sciences and Humanities, a substantially increased citation count is likely for academics who have published the majority of their highly cited work after 1996.

Google Scholar versus ISI and Scopus General Search

As can easily be verified in Table 1, Google Scholar is the most democratic of the three data sources in that it provides the highest level of citations for all ten academics in our sample, with the exception of our Pharmacist.

A. Sciences

Even for the Pharmacist in our sample, Google Scholar only underestimates his citations by 10% and does much better than Scopus in this respect. The underestimation of the Pharmacist’s citation record Google Scholar is most likely caused by Google Scholar’s weaker coverage of older materials, which are not always available on the web. Since our Pharmacist has been publishing for more than 40 years, some of his older work might not be covered in Google Scholar. This assumption is confirmed if we compare ISI and Google Scholars citations to his work in the last decade only. For the last decade alone, citations to his work in Google Scholar are twice as high as ISI citations.

As Figure 3 shows, overall, academics working in the Sciences do not show very significant differences between their ISI and Google Scholar citation scores. For the Pharmacist they are 10% lower, for the Mathematician and Cell Biologist they are only 1-3% higher. For the Physicist, citations in Google Scholar lie 25% above ISI citations.

As before, the Computer Scientist is the exception to this rule. In his case, Google Scholar citations are no less than 7.5 times as high as ISI citations. As for Scopus, this is partly caused by the much broader journal coverage of Google Scholar. However, in this particular case, a very significant chunk of the Google Scholar citations (more than 40%) are citations to a book that – as discussed above – is completely ignored in ISI.

Figure 3: Number of citations for ISI and Scopus General Search compared to Google Scholar: Science disciplines

We have seen above that Scopus citation scores for the Scientists were generally lower than their ISI citation scores. Therefore, it is not surprising that their Google Scholar citation scores are substantially higher than their Scopus citation scores. For the Cell Biologist and the Mathematician Google Scholar citation scores are one third higher than Scopus citation scores, for the Physicist this is two thirds, whilst the Pharmacist has nearly twice as many citations in Google Scholar than in Scopus.

For the Computer Scientist Google Scholar provides five times as many citations as ISI, again reflecting the very significant number of book citations. So overall, although Google Scholar still has a slightly lower coverage of older publications than ISI, it is doing much better than Scopus in this respect.

B. Social Sciences and Humanities

As is readily apparent form Figure 4, for the academics working in the Social Sciences and Humanities, the differences between Google Scholar on the one hand, and ISI or Scopus on the other hand are much larger than for academics working in the Sciences. When comparing Google Scholar and ISI citation scores, the Business academic has six times as many citations in Google Scholar than in ISI.

However, this difference is dwarfed by the increases for the Political Scientist (15 times), the Cinema Studies Academic (25 times), the Linguist (32 times) and the Educationalist (33 times). It is clear that for academics working in the Social Sciences and Humanities, a focus on citations to ISI listed publications only, severely underestimates their citation impact.

Figure 4: Number of citations for ISI and Scopus General Search compared to Google Scholar: Social Science and Humanities disciplines

With regard to the differences between Google Scholar and Scopus, the Social Science and Humanities academics again follow a different pattern from most of the Science academics. Whilst for the Science academics differences between Google Scholar and Scopus are slightly larger than between Google Scholar and ISI, for academics in the Social Sciences and Humanities, they are slightly smaller, but still substantial in absolute terms. Differences run from 3.5 times as many citations in Google Scholar for the Business academic to 27 times as many for the Linguist. The Cinema Studies academic shows a whopping 180 times increase in citations, but that is caused entirely by the very incomplete coverage of her publications in Scopus.

C. Conclusion

Comparing Google Scholar on the one hand and ISI and Scopus on the other hand provides mixed results. For the academics working in the Sciences, Google Scholar’s advantage over Scopus is larger than over ISI (except for the Computer Scientist). For the academics working in the Social Sciences and Humanities, this pattern is reversed in that Google Scholar’s advantage over ISI is larger than over Scopus.

However, in virtually all cases Google Scholar provide the highest citation count, reflecting its broader coverage in terms of sources compared to both ISI and Scopus and its longer coverage in time compared to Scopus.

Publication patterns in the Social Sciences and Humanities include a heavy emphasis on books and many of the journals in these fields are not listed in either ISI or Scopus. Publication patterns in Computer Science include a heavy emphasis on conference proceedings. Google Scholar therefore provides a much fairer assessment of citation impact for academics working in these disciplines. It includes citation in and to books, book chapters, government reports, conference proceedings, as well citations to and in academic journals not listed in ISI or Scopus.

So whilst in the ISI database academics working in the Sciences have on average 17.5 times as many citations as academics working in the Social Sciences and Humanities, in Google Scholar this difference is reduced to a mere 1.5 times. In addition, three of the Social Scientists (Business, Education & Political Science) have citation records that are higher than or comparable with three of the Scientists (Cell Biology, Mathematics, Physics).

ISI Cited by versus ISI General Search

As indicated above, the ISI Cited By search function includes citations to non-ISI listed journals, books, book chapters and conference papers in addition to citations to ISI-listed journals (as in the ISI General search function). The difference between the ISI Cited By search and the ISI General search varies enormously between the various disciplines.

As is apparent from Figure 5, for four of our five academics working in the Sciences, there is virtually no difference between their citation records in the two different types of ISI searches. Their citation records increase only marginally by 0.5% to 4.0%. The simple reason is that in these four disciplines most journals are ISI listed and most academics only publish in journals. In fact, in these four cases the only additional citations in the ISI Cited By search function were caused by “stray citations”, i.e. citations to ISI-listed journals that included either minor misspellings or referred to articles in press, and hence were not correctly merged into the master record by ISI data entry staff.

Figure 5: Number of citations for ISI General and ISI Cited By Search: Science disciplines

The fifth academic, who works in Computer Science, shows a fairly typical citation pattern for Engineering & Computer Science. He has a non-negligible number of additional citations in the ISI Cited By search function, increasing his overall ISI citation record by 31%. In fact, this still seriously underestimates his citation record in ISI-listed journals as it does not include the 700-odd ISI citations to a book for which he is the second author. As indicated above, ISI ignores citations to the second and further author for non-ISI publications. If these were included the Computer Science academic would see his publication record double between the two different types of ISI searches.

In contrast, as is shown in Figure 6, our five academics working in the Social Sciences and Humanities show very large differences between their citation records in the two types of ISI searches. This difference ranges from having nearly 2.5 times as many citations in the IS Cited By search function for the Business and Political Science academic, around 4.5 times as many citations for the Linguist, and 5.5- 6 times as many citations in the ISI Cited By search function for the Education and the Cinemas Studies academic.

Figure 6: Number of citations for ISI General and ISI Cited By Search: Social Science and Humanities disciplines

Consequently, the differences in citation records between the Sciences on the one hand and the Social Sciences and Humanities on the other hand are much larger in the ISI General search function than in the ISI Cited By search function. In the former case, academics in the Sciences on average have 17 times as many citations as the academics in the Social Sciences and Humanities, whilst in the latter case this is reduced to 5 times as many citations.

Scopus More versus Scopus General Search

Scopus also has two search options that are very similar to their ISI equivalent, with the Scopus General Search function only including citations to Scopus-listed journals and the Scopus More function including citations to a wider range of publications. The difference between the two types of Scopus search functions still varies substantially between the various disciplines, but the differences are not as large as for the two types of ISI search functions.

As Figure 7 shows, for two of our five academics working in the Sciences (Cell Biology and Pharmacology), there are very modest differences (1-4%) between their citation records in the two different types of Scopus searches. As for ISI, these were almost entirely caused by “stray citations”, i.e. citations to Scopus listed journals that included either minor misspelling or referred to articles in press and hence were not correctly merged into the master record by Scopus data entry staff.

Figure 7: Number of citations for Scopus General Search and Scopus More Search: Science disciplines

For the Physicist and the Mathematician, the increase is a bit more substantial with 15%-30% additional citations when adding the citations under Scopus More. For the Physicist and the Mathematician these differences are caused by not only by stray citations, but also by Scopus not correctly matching some relatively highly-cited Scopus-listed works to master records.

The Computer Scientist sees his citations double when adding the citations under Scopus More. In addition to stray and incorrectly matched records, this difference is caused largely by citations to a book (not included in the General search). Overall two thirds of the additional citations are citations to a single book.

As Figure 8 shows, differences are generally larger for our academics in the Social Sciences and Humanities, but again vary between the sub-disciplines. The Business academic only gathers 50% more citations, whilst the Education academic, the Linguist and the Political Scientist gather 3 to 4 times as many citations when adding the results of the Scopus More search. The Cinema Studies academic performs very poorly overall in Scopus, which seems to have a serious problem with coverage in this area.

Figure 8: Number of citations for Scopus General Search and Scopus More Search: Social Science and Humanities disciplines

Consequently, the differences in citation records between the Sciences on the one hand and the Social Sciences and Humanities on the other hand are much larger in the Scopus General search function than in the Scopus More search function. In the former case, academics in the Sciences on average have 7.5 times as many citations as the academics in the Social Sciences and Humanities, whilst in the latter case this is reduced to 3.5 times as many citations.

Metrics comparisons across disciplines

Above we have discussed the impact of different data sources on the citation records of our sample of academics in the Sciences and Social Sciences and Humanities. Let's now look at different metrics for our ten professors, using the same database (Google Scholar). All of these metrics were calculated with Publish or Perish, the citation analysis program available from Harzing.com. Table 2 provides a summary of the citation metrics. Below we will discuss each of them in turn.

Table 2: Citation metrics across fields

h-index

Proposed by J.E. Hirsch in his paper An index to quantify an individual's scientific research output, arXiv:physics/0508025 v5 29 Sep 2005. It aims to provide a robust single-number metric of an academic's impact, combining quality with quantity. A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each. Hence an academic with an h-index of 20 has 20 papers with at least 20 citations each.

Our ten professors differ substantially in their h-index, with the Pharmacist having an h-index that is more than three times as high as the Cinema Studies academic. On average the professors in the Sciences have a higher h-index (28) than the professors in the Social Sciences and Humanities (21). However, there are individual Social Sciences academics that have h-indices that are equal to or higher than some of the Science academics.

Number of authors

This metric calculates the average number of co-authors for the academic’s entire body of work by dividing the total number of authors by the total number of papers. As is readily apparent from Figure 9, there is a fairly substantial difference between the Sciences and the Social Sciences and Humanities.

In the Sciences, papers on average have 3 authors (i.e. 2 co-authors), whilst in the Social Sciences and Humanities papers on average only have 1.6 authors (i.e. 0.6 co-author). However, there are substantial differences even within these broader categories. The Computer Scientist on average only has 1.5 co-author, whilst the Cell Biologist on average has nearly three co-authors. The Cinema Studies academic publishes virtually only single-authored work, having only 0.1 co-author on average, whilst the Linguist and the Political Scientist on average have nearly 1 co-author.

Figure 9: Average number of authors per paper in different disciplines

Individual h-index

The Hi-norm corrects for the number of co-authors. We use the Publish or Perish alternative rather than the Batista et al. (2006) option. The latter divides the standard h-index by the average number of authors in the articles that contribute to the h-index.

The PoP alternative first normalizes the number of citations for each paper by dividing the number of citations by the number of authors for that paper, then calculates the h-index of the normalized citation counts. This approach is much more fine-grained than Batista et al.'s; we believe that it more accurately accounts for any co-authorship effects that might be present and that it is a better approximation of the per-author impact, which is what the original h-index set out to provide.

We have seen above that the number of co-authors in the Sciences is generally larger than the number of co-authors in the Social Sciences and Humanities. Hence, we would expect differences between disciplines to be smaller for the individual h-index than for the general h-index. Figure 10 shows that this is indeed the case.

At 28 the average h-index in the Sciences is one third higher than the average h-index in the Social Sciences and Humanities at 21. However, the average Hi-norm for the two groups is virtually identical: 17 for the Sciences and 18 for the Social Sciences and Humanities.

Figure 10: H-index compared with the Individual h-index for different disciplines

Contemporary h-index

The Hc-index corrects for the recentness of the citations, with recent citations carrying more weight. It was proposed by Antonis Sidiropoulos, Dimitrios Katsaros, and Yannis Manolopoulos in their paper Generalized h-index for disclosing latent facts in citation networks, arXiv:cs.DL/0607066 v1 13 Jul 2006. It adds an age-related weighting to each cited article, giving (by default; this depends on the parametrization) less weight to older articles.

The weighting is parametrized; the Publish or Perish implementation uses gamma=4 and delta=1, like the authors did for their experiments. This means that for an article published during the current year, its citations account four times. For an article published 4 years ago, its citations account only one time. For an article published 6 years ago, its citations account 4/6 times, and so on.

As the number of years that our academics have been active varies from 16 to 42, we would expect differences between them to be smaller for the Hc-index than for the h-index. Figure 11 shows that this is indeed the case. Our pharmacologist has been active for 42 and has the highest h-index. However, much of high highly-cited work was published a long time ago and his Hc-index is less than half of his regular h-index.

In contrast, the Business and Education academic have been active only 16 and 28 years and have recently published work that is highly cited. Hence their Hc-indices are nearly 90% of their h-indices. Differences between the Sciences and the Social Sciences and Humanities are smaller for the Hc-index (17 versus 16) than for the h-index (28 versus 21).

Figure 11: H-index compared with the Contemporary index for academics in different disciplines

Hirsch’s m and Individual m

Proposed by J.E. Hirsch in his paper An index to quantify an individual's scientific research output, arXiv:physics/0508025 v5 29 Sep 2005. It is calculated by dividing the h-index by the number of years the academic has been active. The latter is defined as the number of years that have passed since publication of the first paper. The Individual m is calculated by dividing the individual h-index by the number of years the academic has been active.

In his article Hirsch proposes norm scores for m for Physicists. He suggests that academics with an m of 1 can be considered to be successful academics, whilst academics with an m of 2 or 3 can be considered respectively as outstanding or truly unique individuals. To put this into context, consider that a popular physicist such as Stephen Hawking has an m of “only” 1.59. I would therefore suggest that we should consider an m of around 1 to reflect excellence rather than simple success in terms of research impact.

As can be seen in Table 2 and Figure 12, on average academics in both general fields have a very similar Hirsch’s m index, with 0.95 for the Sciences and 0.93 for the Social Sciences and Humanities. This suggests that on average professors at the University of Melbourne should be considered to display excellence in terms of research impact.

In fact, most of the individual academics hover around the m=1.0 mark, with only the Mathematician and the Cinema Studies academics scoring substantially lower and the Business academic scoring substantially higher. The latter case might be slightly anomalous as this academic has only been active for 16 years. She might not be able to maintain this level of performance over the next decades.

Looking at the individual m-index, which corrects for the number of co-authorships we see that on average academics in the Social Sciences and Humanities perform better than academics in the Sciences as their individual m-index is only 20% lower than their m-index, whilst academics in the Sciences experience nearly a 40% drop.

Figure 12: Hirsch’s m and Individual m for academics in different disciplines

Conclusion

As a Social Scientist, my single biggest concern with the focus on research metrics is that ISI citations drawn from the ISI General search function (and their derivatives, such as the ISI h-index) are still the most commonly used metric. As the analysis in this white paper shows, this seriously disadvantages academics in both Engineering-/Computer Science and the Social Sciences and Humanities as it underestimates citations records in these fields.

We started out our story with our professors in Science out-performing our professors in the Social Sciences and Humanities to a staggering extent, by having 17 times as many ISI citations. At the end of our story, we find that when using the most comprehensive data source and correcting for the number of co-authors and the length of the academic’s publishing career, academics in the Social Sciences and Humanities on average out-perform academics in the Sciences.

The underestimation of research impact in the Social Sciences and Humanities might be even more pronounced in terms of research effort. A typical article in the Social Sciences and Humanities is 20 pages long and often requires three time-consuming rounds of revisions before it is accepted for a major journal. A typical paper in Medicine/Sciences is only 2-5 pages long and requires fewer revisions before it is accepted for a journal.

In addition, many academics in the Social Sciences and Humanities publish books that might run to 500 pages (still counting for only one publication). In fact, all of our Social Science and Humanities academics had published at least two books, which were generally amongst their most-cited works. Hence, even when one corrects for the number of co-authors, it is still not fair to compare the number of publications across disciplines.

On the other hand, one could argue that research in the Sciences generally requires larger amounts of funding. Hence academics in the Sciences might be forced to spend a large part of their time writing up grant applications rather than writing up articles, thus limiting their output.

So the inescapable conclusion is that one should not attempt to compare research performance across disciplines. However, I argue that the current emphasis on research metrics and the fact that most University administrators only know the ISI General search function seriously disadvantages the Social Sciences and Humanities. This paper has established that with the correct benchmarks academics in the Social Sciences and Humanities perform at least as well, if not better, than academics in the Sciences.

Summary

So to cut a long story short:

• Academic performance in terms of citation impact can differ enormously depending on the data source and metrics used.
• It is not normally appropriate to compare performance across disciplines. If one has to do so, Google Scholar is the most appropriate data source.
• The Hi-index and Hc-index (or Hirsch’s m or Individual m) are more appropriate metrics to compare academics across disciplines and career stages than the traditional h-index.
• Academics in the Sciences out-perform academics in the Social Sciences and Humanities by a factor of 17.5 if one considers the traditional performance indicator: ISI General Search citations.
• Academics in the Social Sciences and Humanities out-perform academics in the Sciences when using a more comprehensive data-source and correcting for career stage and the number of co-authors.

References

Batista, P.D.; Campiteli, M.G.; Konouchi, O.; Martinez, A.S. (2006) Is it possible to compare researchers with different scientific interests? Scientometrics, vol. 68, no. 1, pp. 179-189.

Harzing, A.W.; Wal, R. van der (2008) Google Scholar as a new source for citation analysis?, Ethics in Science and Environmental Politics, vol. 8, no. 1, pp. 62-71

Vaughan, L.; Shaw, D. (2008) A new look at evidence of scholarly citations in citation indexes and from web sources, Scientometrics, vol. 74, no. 2., pp. 317-330.

Find the resources on my website useful?

I cover all the expenses of operating my website privately. If you enjoyed this post and want to support me in maintaining my website, consider buying a copy of one of my books (see below) or supporting the Publish or Perish software.

Aug 2022: Only £5.95...

Nov 2022: Only £5.95...

Feb 2023: Only £5.95...

May 2023: Only £5.95...

citation analysis citation metrics google scholar white papers

Reddit Twitter LinkedIn Facebook Email