The Publish or Perish Book Also available
in paperback		       Publish or Perish logo
Home | ToC | Index   < Previous | Next >

Chapter 15: A Google Scholar h-index for journals > 15.3 Results and discussion of the benchmarking analysis > 15.3.3 Analysis of individual sub-disciplines

15.3.3 Analysis of individual sub-disciplines

The Journal Quality List includes journals in fifteen different sub-disciplines. However, for some of these sub-disciplines only a small number of journals are included, either because the sub-discipline is a very specialized area (e.g. Innovation, Entrepreneurship, Tourism) or because the Journal Quality List only includes a small subset of journals in the sub-discipline in question (e.g. Psychology/Sociology) that are relevant to Economics & Business. Overall, there are seven sub-disciplines that have a substantial number (more than 60) journals included in the JQL. These seven sub-disciplines cover 75% of the journals in the Journal Quality List.

Table 2 provides some summary statistics for these seven sub-disciplines and show there is significant variability in terms of the proportion of ISI-indexed journals in the different fields, ranging from a low of 30-43% for Finance & Accounting, Marketing and General Management & Strategy to a high of 74-80% for Economics, Management Information Systems and Management Science & Operations Research/Management.

The sub-disciplines also differ in terms of the strength of correlation between the h-index and the JIF, varying from 0.633 for Organization Behaviour/Studies; Human Resource Management & Industrial Relations to 0.891 for General Management & Strategy, but in all cases this correlation was significant at p < 0.001. Below I provide a detailed benchmarking analysis for each of these sub-disciplines.

Table 2: Summary statistics

Sub-field No. of journals in the JQL No. of ISI-indexed journals Spearman correlation b/w h-index & JIF
Economics 158 122 (74%) 0.732***
Finance & Accounting 94 28 (30%) 0.721***
General Management & Strategy 63 27 (43%) 0.891***
Mgmt Information Systems & Knowledge Mgmt 81 61 (75%) 0.774***
Mgmt Science; Operations Research & Mgmt 87 70 (80%) 0.733***
Marketing 65 25 (38%) 0.841***
Org. Behaviour/Studies; HRM & Industrial Relations 71 45 (63%) 0.633***
Others 209 158 (76%) 0.764***
Total 838 536 (64%) 0.718***

***p < 0.001

Economics

Figure 3 (see the end of this chapter) shows the relationship between the Google Scholar h-index and the JIF for Economics journals. Most journals cluster around the regression line. Two important outliers that show higher journal impact factors than h-indices are the Journal of Economic Literature and the Quarterly Journal of Economics; both have a JIF above 4.5, but modest h-indices.

The other outliers are less prominent, though we can distinguish a number of health economics journals that are likely to display the high immediacy of Science journals and hence fare better on the 2-year JIF.

On the other side of the spectrum are Review of Economics & Statistics, Research Policy and European Economic Review that have a relatively high Google Scholar h-index in comparison to their more modest ISI JIF. The main reason for this appears to be that all three journals show a large number of citations in working papers and policy documents, or journals not covered by ISI. Hence the h-index captures their significant impact beyond academic journals.

Finance & Accounting

As Table 2 shows there are 94 journals in the Finance & Accounting category (93 excluding the Journal of Finance), but only 28 have both an ISI JIF and a Google Scholar h-index; only 30% of the Finance & Accounting journals listed in the JQL are ISI indexed. The correlation between the ISI JIF and the GS h-index for these journals is slightly higher than average at 0.721 (p < 0.001). Figure 4 (see the end of this chapter) shows that although many journals cluster close to the regression line, there are a number of significant outliers.

Important outliers on the right-hand side (i.e. higher h-index than JIF) are: Journal of Money Credit & Banking, Journal of Banking & Finance, Journal of International Money & Finance and International Journal of Finance & Economics. Papers in these journals often deal with issues relating to stock markets, credit rating and exchange rates and tend to be cited quite often in working papers (e.g. from the National Bureau of Economic Research) and policy documents (e.g. from the Federal Reserve Bank), or in journals not covered by ISI. As a result their Google Scholar h-index is much higher than their JIF that only measures impact in academic journals listed in ISI.

General Management & Strategy

As Table 2 shows out of the 63 journals in the General Management & Strategy category, there are only 27 (43%) that have both and ISI and a Google Scholar Ranking. However, for those journals that are ISI indexed the correlation between the JIF and their GS h-index is 0.891 (p < 0.001), the highest of all sub-disciplines.

It is therefore not surprising that there are relatively few important outliers in Figure 5 (see the end of this chapter). The main outliers on the left-hand side (high JIF in comparison to h-index) are two of the absolute top journals in the field of Management: Administrative Science Quarterly and Academy of Management Review, which have the highest impact factors of any journal in General Management & Strategy, but have a relatively lower h-index.

Administrative Science Quarterly

With regard to Administrative Science Quarterly, this appears to be caused mostly by the limited yearly number of papers published. Even though most of the articles published in this journal are fairly well-cited, Administrative Science Quarterly only published a total of 92 papers (excluding editorials and book reviews) in the 5-year period and hence its ability to achieve a very high h-index is limited.

One should also consider that even in a top journal such as Administrative Science Quarterly, the citations received by individual articles are highly skewed: the ten most highly cited papers received 30% of the total citations, whilst the twenty most cited papers received 50% of the total number of citations.

Journal of Management, Journal of Management Studies and Journal of International Business Studies show h-indices comparable to Administrative Science Quarterly, even though they are generally seen to be lower in standing. However, these journals publish about two (for Journal of Management and Journal of International Business Studies) to three times (for Journal of Management Studies) as many papers per year as Administrative Science Quarterly and hence have a higher likelihood of reaching a high h-index.

Academy of Management Review

With regard to the Academy of Management Review, citations are even more heavily skewed than for Administrative Science Quarterly. The top 4 most cited papers (dealing with key concepts such as social capital, absorptive capacity and the resource-based view (2x)) provide 21% of the total number of citations. Surpassing even Administrative Science Quarterly, the 10 most cited papers provide 34% of the total citations and again the twenty (out of 153) most cited papers received 50%.

Hence even though at 46 Academy of Management Review has one of the highest h-indices for General Management journals, its concentrated citation pattern means that its h-index is low in comparison to its JIF. And even though with around 150 papers (excluding editorials and book reviews) over five years it publishes more articles per year than ASQ, its empirical counterpart (Academy of Management Journal) publishes twice as many papers as Academy of Management Review and hence has a higher likelihood of reaching a high h-index.

Furthermore, more than half of the Academy of Management Review papers are classified as either editorials or book reviews. Citations to these non-source materials are included in the numerator of the JIF, but the non-source materials are not included in the denominator. Normally, this would not result in a significant distortion of the JIF as these non-source materials tend not to be highly cited in the field of Management, but the paper-length introductions to the many special issues and forums are also classified as editorials and these pieces tend to be highly cited.

Other journals

The larger number of papers published (200-550 papers over 5 years) is also likely to lie behind the relatively high h-index for Strategic Management Journal, Harvard Business Review, Sloan Management Review, Journal of Business and Journal of Management Studies.

Further, the two practitioner journals (Harvard Business Review and Sloan Management Review) are also likely to be more highly cited in scholarly policy documents that are not incorporate in ISI. The same is likely to be true for Journal of Business that has many papers that would be cited in working papers (e.g. from the National Bureau of Economic Research) and policy documents (e.g. from the Federal Reserve Bank). Further, the UK-based Journal of Management Studies is likely to be more heavily cited in non-ISI listed European journals.

Harzing and Van der Wal (2008) also showed that Strategic Management Journal scores much better in Google Scholar than the general management journals, because many Strategy and IB journals that would heavily cite articles in SMJ are not ISI listed. As a result Strategic Management Journal has a higher h-index than Academy of Management Journal even though its JIF is lower.

Management Information Systems; Knowledge Management

As can be seen in Figure 6 (see the end of this chapter), the various ACM Transactions (on Data Base Systems, on Software Engineering and Methodology, on Information Systems) generally have low h-indices compared to their JIF. The same is true for Information Systems, Information Systems Research, Human Computer Interaction, MIS Quarterly and the Journal of Database Management.

The various IEEE Transactions and Communications of the ACM have a high h-index in comparison to their relatively modest JIF. This is partly due to the fact that articles in these journals are often cited in conference proceedings, which are the most important publication outlets in this field, but are not included in the ISI citation count. However, this is true to a large extent for the ACM Transactions as well.

The main difference between the two groups of journals is the number of articles they publish. For the various IEEE Transactions this lies in the 350-600 range for a five-year period and for IEEE Transactions on Automatic Control even exceeds 1400, while for Communications of the ACM it approaches 1000. As a result, JIFs do not fluctuate as widely as for the other group of journals as they are not influenced by individual highly cited papers. On the other hand, the larger publication base makes it easier to achieve high h-indices, reflecting these journals substantial impact on the field.

Management Science; Operations Research, Operations Management

As Table 2 shows out of the 87 (86 without SIAM Review) MS/OR/POM journals in the Journal Quality List 70 (80%) are ISI indexed. The correlation between the ISI JIF and the GS h-index for these journals lies slightly above the average at 0.733 (p < 0.001). and is highly significant.

As Figure 7 (see the end of this chapter) shows, both of the journals of the Royal Statistical Society as well as Annals of Statistics have relatively high JIFs in comparison to their Google Scholar h-index. This is caused by the fact that even though they publish a reasonably large number of papers, citations are highly concentrated.

For JRSS-A and Annals of Statistics 25% of the citations go to the top 5 most cited papers (out of some 150), for JRSS-B it is even only the two most cited papers (out of some 235) that make up 25% of total citations, with top-10 most cited papers making up 50% of the total number of citations. As we have seen before, a concentrated citation score will always artificially inflate the JIF and lead to a lower h-index in comparison to the JIF.

At the other end of the spectrum Management Science, European Journal of Operational Research and Operations Research have Google Scholar h-indices that are relatively high in comparison to their JIF.

As a result the JIFs for these journals are generally very stable and not influenced by individual highly-cited papers. The large number of papers and even spread of citations ensures a high h-index, reflecting these journals substantial impact on the field.

Marketing

As Table 2 shows there are only 25 journals in the Marketing category that have both and ISI and a Google Scholar Ranking, 40% of the 65 Marketing journals in the Journal Quality List. However, for those journals that are ISI indexed the correlation between the JIF and their Google Scholar h-index is 0.841 (p < 0.001), the second highest of all sub-disciplines. It is therefore not surprising that there are relatively few important outliers in Figure 8 (see the end of this chapter).

The two top journals in Marketing, Marketing Science and Journal of Marketing, have JIFs that are relatively high in comparison to their Google Scholar h-index.

Industrial Marketing Management and Journal of Business Research are the two most important outliers at the other end of the spectrum with a relatively high Google Scholar h-index in comparison to their modest JIF. Both journals publish a relatively large number of papers (350 to 600 over five years) increasing their chances of reaching a high h-index. They are also cited quite often in journals that are not ISI indexed, such as European Journal of Marketing, Journal of Business to Business Marketing and Journal of Business & Industrial Marketing, which increases their Google Scholar h-index over their JIF and better reflects their overall impact on the field of marketing.

Organization Behaviour/Studies; HRM & Industrial Relations

At 0.633 (p < 0.001) this sub-discipline has the lowest correlation between the JIF and the GS h-index, but it is still high and very significant. Excluding the Human Resource Management and Journal of Organizational Behavior Management outliers (see below) raises this correlation to 0.685 (p < 0.001).

As Figure 9 (see the end of this chapter) shows several journals have a high JIF in relation to their Google Scholar h-index. The most striking cases are Human Resource Management and Journal of Organizational Behavior Management.

Human Resource Management

Harzing & Van der Wal (2008) already noted the discrepancy between Human Resource Managements JIF and h-index. Further investigation and an extensive email exchange with Thomson ISI representatives revealed that Thomsons search query for this journals JIF included a substantial number of homographs referring to Human Resource Management Review, Human Resource Management Journal as well as books with Human Resource Management in their title.

As a result the JIF for Human Resource Management was erroneously inflated. In fact at 0.64 the recently released 2007 JIF for Human Resource Management is very substantially lower than the 2002-2006 JIF average of 2.00.

Journal of Organizational Behavior Management

The JIF for Journal of Organizational Behavior Management has fluctuated wildly, standing at for instance 1.79 in 2003 and 0.11 in 2004. In 2004 there were only two citations to articles published in 2003 and 2002. In 2003 there were no less than 52 citations to articles published in 2002 and 2001. All of the 34 citations to JOBM in 2002 came from JOBM articles (i.e. a 100% self-citation on a journal level), whilst 11 of the 18 citations to articles published in 2001 came from within the journal. Further investigation showed that a very large proportion of the citations to papers in 2002 consisted of within-issue citations in a special issue. The third and fourth most cited of all papers in JOBM between 2001 and 2005 were published in this issue.

The journals relatively low Google Scholar h-index is due to Google Scholar's less than complete records for this journal. Google Scholar appears to have processed the papers published in the special issue, but does not seem to have parsed the citations in these papers. In this case, however, I would argue that this is probably for the best and even though Google Scholar is to some extent at fault, it provides a more realistic assessment of this journals low impact beyond a seemingly rather small academic circle.

Other journals

On the other end of the spectrum we find journals that have a high h-index in comparison to their JIF. Journal of Human Resources publishes many policy oriented pieces dealing with the public sector that are heavily cited in working papers and policy papers as well as books not included in ISI. As a result the h-index better reflects its impact beyond academia.

The Journal of Business Ethics, Human Relations and International Journal of Human Resources Management, all published out of Europe, combine a relatively large number of published papers with citations that are evenly spread across papers, as well as a high number of citations in European journals not indexed in ISI, all three of which increase their Google Scholar h-index in comparison to their JIF. Hence the h-index provides a more accurate reflection of their relatively broad impact on the field.

Overall conclusions about sub-discipline analysis

Overall, we have seen that there is a very substantial agreement between the ISI JIF and the Google Scholar h-index for most sub disciplines. This means that for those sub-disciplines that have very limited ISI coverage (Finance & Accounting, Marketing and General Management & Strategy) the Google Scholar h-index could provide an excellent alternative for the 56-70% of journals not covered in ISI, especially given that for these disciplines the correlation between JIF and h-index for journals that did have ISI coverage ranged between .72 and .89. However, even for the other sub-disciplines the additional coverage provided by Google Scholar could be useful. Where the ISI JIF and the Google Scholar h-index diverged this was generally caused by one of two factors.

Factor 1: JIF's sensitivity to one-hit wonders

The sensitivity of the JIF to individual highly cited papers, which - especially for journals that publish relatively few papers artificially inflates the JIF in comparison to the h-index. Using mean scores produces distorted results if distributions are non-normal. In this respect, the h-index is a more robust measure. It is true that the h-index is influenced by the number of papers published and hence that journals that publish a lot of papers have a better chance to reach a high h-index. However, journals producing a larger number of highly cited papers have more impact on the field even if the average article in this journal is less highly cited than the average article in a journal that publishes fewer articles.

Journals such as Strategic Management Journal, the Academy of Management Journal, Organization Science and Management Science have higher h-indices than the Journal of Marketing and the Academy of Management Review (on average 55 versus on average 46), whereas they have much lower JIFs (on average 2.32 versus on average 3.95). However, as we have seen above, the high JIFs of the latter two journals are mostly caused by a very limited number (2-4) highly cited papers. Hence, I would argue that the h-index provides a more accurate picture of the generally very similar standing of these six journals.

Factor 2: Google Scholar's broader coverage

The key reason why some journals showed a relatively high h-index in comparison to their JIF lies in the broader coverage of Google Scholar. There are four main aspects to this broader coverage.

Overall, I therefore argue that the Google Scholar h-index provides a more comprehensive measure of journal impact, in terms of both the number of journals and the number of citations covered.

Home | ToC | Index   < Previous | Next >
The Publish or Perish Book © 2010-2013 Anne-Wil Harzing. All rights reserved. $Id: ch15_3_3.htm 5171 2016-01-26 17:40:58Z dave $