15.3.3 Analysis of individual sub-disciplines
The Journal Quality List includes journals in fifteen different sub-disciplines. However, for some of these sub-disciplines only a small number of journals are included, either because the sub-discipline is a very specialized area (e.g. Innovation, Entrepreneurship, Tourism) or because the Journal Quality List only includes a small subset of journals in the sub-discipline in question (e.g. Psychology/Sociology) that are relevant to Economics & Business. Overall, there are seven sub-disciplines that have a substantial number (more than 60) journals included in the JQL. These seven sub-disciplines cover 75% of the journals in the Journal Quality List.
- Finance & Accounting
- General Management & Strategy
- Management Information Systems & Knowledge Management
- Management Science/Operations Research/Production & Operations Management
- Organization Studies/Behavior; Human Resource Management & Industrial Relations.
Table 2 provides some summary statistics for these seven sub-disciplines and show there is significant variability in terms of the proportion of ISI-indexed journals in the different fields, ranging from a low of 30-43% for Finance & Accounting, Marketing and General Management & Strategy to a high of 74-80% for Economics, Management Information Systems and Management Science & Operations Research/Management.
The sub-disciplines also differ in terms of the strength of correlation between the h-index and the JIF, varying from 0.633 for Organization Behaviour/Studies; Human Resource Management & Industrial Relations to 0.891 for General Management & Strategy, but in all cases this correlation was significant at p < 0.001. Below I provide a detailed benchmarking analysis for each of these sub-disciplines.
Table 2: Summary statistics
|Sub-field||No. of journals in the JQL||No. of ISI-indexed journals||Spearman correlation b/w h-index & JIF|
|Finance & Accounting||94||28 (30%)||0.721***|
|General Management & Strategy||63||27 (43%)||0.891***|
|Mgmt Information Systems & Knowledge Mgmt||81||61 (75%)||0.774***|
|Mgmt Science; Operations Research & Mgmt||87||70 (80%)||0.733***|
|Org. Behaviour/Studies; HRM & Industrial Relations||71||45 (63%)||0.633***|
***p < 0.001
Figure 3 (see the end of this chapter) shows the relationship between the Google Scholar h-index and the JIF for Economics journals. Most journals cluster around the regression line. Two important outliers that show higher journal impact factors than h-indices are the Journal of Economic Literature and the Quarterly Journal of Economics; both have a JIF above 4.5, but modest h-indices.
- Journal of Economic Literature publishes a relatively small number of articles per year (15-20), so that even though most of these are highly cited, it will be difficult for the journal to achieve a very high h-index. This is almost an exact counter case to the American Economic Review, which publishes around 160-170 articles per year that on average are not as highly cited as articles in the Journal of Economic Literature. Overall, however, the American Economic Review has a much larger total number of articles that are highly cited and I therefore argue that the h-index correctly identifies its more substantial contribution to the field of Economics.
- In the case of the Quarterly Journal of Economics, it seems that Google Scholar is at fault as it misses a number of highly cited papers in this journal. Its automatic parsing mechanism seems to have misclassified them under the wrong journal. For instance "Understanding social preferences with simple tests" by Charness and Rabbin (2002) is assigned to the (non-existing?) journal Technology. Several other papers are listed under their earlier (and highly cited?) publications as a NBER working paper. In this case it is clear we should ignore the Google Scholar results and give preference to the ranking presented by the ISI JIF.
The other outliers are less prominent, though we can distinguish a number of health economics journals that are likely to display the high immediacy of Science journals and hence fare better on the 2-year JIF.
- The Journal of Economic Geography and the Journal of Economic Growth also publish a relatively small number of papers (15-25/year) and hence their JIFs are quite heavily influenced by a rather small number of highly cited papers. As a result their h-index is relatively low in comparison to their JIF.
- The JIFs for Journal of Economic Geography for 2004 and 2005 also seem to have been inflated by a highly cited editorial. As explained above citations to editorial materials and book reviews are included in the numerator of the JIF, but they are not included in the denominator, thus artificially inflating the JIF.
- Demography publishes a larger number of papers (app. 200 in 5 years), but also has a highly concentrated citation pattern, with its most cited paper between 2001 and 2005 taking up nearly 10% of total citations.
On the other side of the spectrum are Review of Economics & Statistics, Research Policy and European Economic Review that have a relatively high Google Scholar h-index in comparison to their more modest ISI JIF. The main reason for this appears to be that all three journals show a large number of citations in working papers and policy documents, or journals not covered by ISI. Hence the h-index captures their significant impact beyond academic journals.
Finance & Accounting
As Table 2 shows there are 94 journals in the Finance & Accounting category (93 excluding the Journal of Finance), but only 28 have both an ISI JIF and a Google Scholar h-index; only 30% of the Finance & Accounting journals listed in the JQL are ISI indexed. The correlation between the ISI JIF and the GS h-index for these journals is slightly higher than average at 0.721 (p < 0.001). Figure 4 (see the end of this chapter) shows that although many journals cluster close to the regression line, there are a number of significant outliers.
- Two important outliers on the left-hand side (i.e. higher JIF than h-index) are Journal of Accounting & Economics and Review of Accounting Studies.
- Journal of Accounting & Economics has rather variable JIFs. In 2001-2002 and 2004-2005 its average JIF was around 1.7 which would place it very close to the regression line. However, in both 2003 and 2006 its impact factor more than doubled. Reviewing the individual articles revealed a small number of highly cited papers in 2001 and 2005, which given the limited number of papers published yearly in this journal have a significant impact on its JIF.
- Review of Accounting Studies was only ISI indexed recently and only has a JIF for two years (2005/ 2006). Its JIF for 2006 is substantially higher than that for 2005. A review of individual articles published in 2005 showed one highly cited paper (The role of analysts' forecasts in accounting-based valuation: A critical evaluation by Q Cheng) that with 78 cites in October 2007 made up nearly two thirds of the total citations to articles in 2005; the remaining fifteen articles on average had only 3 cites each. Again a concentrated citation pattern, combined with a small number of published papers (51 in 5 years) results in a high JIF without a similar impact on the h-index.
Important outliers on the right-hand side (i.e. higher h-index than JIF) are: Journal of Money Credit & Banking, Journal of Banking & Finance, Journal of International Money & Finance and International Journal of Finance & Economics. Papers in these journals often deal with issues relating to stock markets, credit rating and exchange rates and tend to be cited quite often in working papers (e.g. from the National Bureau of Economic Research) and policy documents (e.g. from the Federal Reserve Bank), or in journals not covered by ISI. As a result their Google Scholar h-index is much higher than their JIF that only measures impact in academic journals listed in ISI.
General Management & Strategy
As Table 2 shows out of the 63 journals in the General Management & Strategy category, there are only 27 (43%) that have both and ISI and a Google Scholar Ranking. However, for those journals that are ISI indexed the correlation between the JIF and their GS h-index is 0.891 (p < 0.001), the highest of all sub-disciplines.
It is therefore not surprising that there are relatively few important outliers in Figure 5 (see the end of this chapter). The main outliers on the left-hand side (high JIF in comparison to h-index) are two of the absolute top journals in the field of Management: Administrative Science Quarterly and Academy of Management Review, which have the highest impact factors of any journal in General Management & Strategy, but have a relatively lower h-index.
Administrative Science Quarterly
With regard to Administrative Science Quarterly, this appears to be caused mostly by the limited yearly number of papers published. Even though most of the articles published in this journal are fairly well-cited, Administrative Science Quarterly only published a total of 92 papers (excluding editorials and book reviews) in the 5-year period and hence its ability to achieve a very high h-index is limited.
One should also consider that even in a top journal such as Administrative Science Quarterly, the citations received by individual articles are highly skewed: the ten most highly cited papers received 30% of the total citations, whilst the twenty most cited papers received 50% of the total number of citations.
Journal of Management, Journal of Management Studies and Journal of International Business Studies show h-indices comparable to Administrative Science Quarterly, even though they are generally seen to be lower in standing. However, these journals publish about two (for Journal of Management and Journal of International Business Studies) to three times (for Journal of Management Studies) as many papers per year as Administrative Science Quarterly and hence have a higher likelihood of reaching a high h-index.
Academy of Management Review
With regard to the Academy of Management Review, citations are even more heavily skewed than for Administrative Science Quarterly. The top 4 most cited papers (dealing with key concepts such as social capital, absorptive capacity and the resource-based view (2x)) provide 21% of the total number of citations. Surpassing even Administrative Science Quarterly, the 10 most cited papers provide 34% of the total citations and again the twenty (out of 153) most cited papers received 50%.
Hence even though at 46 Academy of Management Review has one of the highest h-indices for General Management journals, its concentrated citation pattern means that its h-index is low in comparison to its JIF. And even though with around 150 papers (excluding editorials and book reviews) over five years it publishes more articles per year than ASQ, its empirical counterpart (Academy of Management Journal) publishes twice as many papers as Academy of Management Review and hence has a higher likelihood of reaching a high h-index.
Furthermore, more than half of the Academy of Management Review papers are classified as either editorials or book reviews. Citations to these non-source materials are included in the numerator of the JIF, but the non-source materials are not included in the denominator. Normally, this would not result in a significant distortion of the JIF as these non-source materials tend not to be highly cited in the field of Management, but the paper-length introductions to the many special issues and forums are also classified as editorials and these pieces tend to be highly cited.
The larger number of papers published (200-550 papers over 5 years) is also likely to lie behind the relatively high h-index for Strategic Management Journal, Harvard Business Review, Sloan Management Review, Journal of Business and Journal of Management Studies.
Further, the two practitioner journals (Harvard Business Review and Sloan Management Review) are also likely to be more highly cited in scholarly policy documents that are not incorporate in ISI. The same is likely to be true for Journal of Business that has many papers that would be cited in working papers (e.g. from the National Bureau of Economic Research) and policy documents (e.g. from the Federal Reserve Bank). Further, the UK-based Journal of Management Studies is likely to be more heavily cited in non-ISI listed European journals.
Harzing and Van der Wal (2008) also showed that Strategic Management Journal scores much better in Google Scholar than the general management journals, because many Strategy and IB journals that would heavily cite articles in SMJ are not ISI listed. As a result Strategic Management Journal has a higher h-index than Academy of Management Journal even though its JIF is lower.
Management Information Systems; Knowledge Management
As can be seen in Figure 6 (see the end of this chapter), the various ACM Transactions (on Data Base Systems, on Software Engineering and Methodology, on Information Systems) generally have low h-indices compared to their JIF. The same is true for Information Systems, Information Systems Research, Human Computer Interaction, MIS Quarterly and the Journal of Database Management.
- For the ACM Transactions this is most likely caused by the fact that they publish relatively few papers (between 60 and 90 over 5 years). This means on the one hand that an individual highly cited paper can substantially increase the JIF for certain years and on the other hand that it is more difficult to achieve a high h-index.
- The same is true for Human Computer Interaction which has highly variable JIFs ranging from 1.95 to 4.78. Again, a small number of published papers (60, excluding editorials), combined with individual highly-cited outliers and a substantial proportion of non-source materials (editorials) which account for 10% of the citation count, drive up the JIF, whilst not having the same impact on the h-index.
- Information Systems, Information Systems Research and MIS Quarterly also have widely varying JIFs (0.90-3.33, 1.17-3.51 and 1.80-4.98 respectively) and although they publish a larger number of papers than HCI, the number of published papers is still relatively small (100-150) and individual highly-cited outliers as well as highly cited editorials can still have a substantial impact on the JIF in some years. In MIS Quarterly for example, the top 4 most cited papers published between 2001-2005 (out of 138 papers) make up 25% of the total number of citations in October 2007.
- The Journal of Database Management has only been ISI-indexed in 2006 and hence its high JIF score might be idiosyncratic. It also publishes few papers (about 15 a year) and hence its ability to achieve a high h-index is limited.
The various IEEE Transactions and Communications of the ACM have a high h-index in comparison to their relatively modest JIF. This is partly due to the fact that articles in these journals are often cited in conference proceedings, which are the most important publication outlets in this field, but are not included in the ISI citation count. However, this is true to a large extent for the ACM Transactions as well.
The main difference between the two groups of journals is the number of articles they publish. For the various IEEE Transactions this lies in the 350-600 range for a five-year period and for IEEE Transactions on Automatic Control even exceeds 1400, while for Communications of the ACM it approaches 1000. As a result, JIFs do not fluctuate as widely as for the other group of journals as they are not influenced by individual highly cited papers. On the other hand, the larger publication base makes it easier to achieve high h-indices, reflecting these journals substantial impact on the field.
Management Science; Operations Research, Operations Management
As Table 2 shows out of the 87 (86 without SIAM Review) MS/OR/POM journals in the Journal Quality List 70 (80%) are ISI indexed. The correlation between the ISI JIF and the GS h-index for these journals lies slightly above the average at 0.733 (p < 0.001). and is highly significant.
As Figure 7 (see the end of this chapter) shows, both of the journals of the Royal Statistical Society as well as Annals of Statistics have relatively high JIFs in comparison to their Google Scholar h-index. This is caused by the fact that even though they publish a reasonably large number of papers, citations are highly concentrated.
For JRSS-A and Annals of Statistics 25% of the citations go to the top 5 most cited papers (out of some 150), for JRSS-B it is even only the two most cited papers (out of some 235) that make up 25% of total citations, with top-10 most cited papers making up 50% of the total number of citations. As we have seen before, a concentrated citation score will always artificially inflate the JIF and lead to a lower h-index in comparison to the JIF.
At the other end of the spectrum Management Science, European Journal of Operational Research and Operations Research have Google Scholar h-indices that are relatively high in comparison to their JIF.
- Management Science publishes a large number of papers (nearly 600 over five years) and has a very evenly spread citation pattern with the top 10 most highly cited papers in ISI in October 2007 having a very similar number of citations (between 41 and 55) and making up less than 10% of the total number of citations.
- European Journal of Operational Research publishes even more papers (nearly 2000 over five years) and show a similarly even spread in citations, with the top 10 most highly cited papers making up less than 5% of the total number of citations.
- Operations Research publishes fewer papers than the other two (but still nearly 400 over five years), but again has a very evenly spread citation pattern, with the top 10 most highly cited papers all having between 23 and 29 citations for a total of 13% of the total number of citations.
As a result the JIFs for these journals are generally very stable and not influenced by individual highly-cited papers. The large number of papers and even spread of citations ensures a high h-index, reflecting these journals substantial impact on the field.
As Table 2 shows there are only 25 journals in the Marketing category that have both and ISI and a Google Scholar Ranking, 40% of the 65 Marketing journals in the Journal Quality List. However, for those journals that are ISI indexed the correlation between the JIF and their Google Scholar h-index is 0.841 (p < 0.001), the second highest of all sub-disciplines. It is therefore not surprising that there are relatively few important outliers in Figure 8 (see the end of this chapter).
The two top journals in Marketing, Marketing Science and Journal of Marketing, have JIFs that are relatively high in comparison to their Google Scholar h-index.
- Marketing Science has experienced an important increase in its JIF over recent years, from an average of 1.90 between 2001 and 2003, which would locate it close to the regression line, to an average of 3.72 between 2004 and 2006. This increase seems to have been caused by a dozen of well-cited papers, few of which, however, were cited often enough to become part of the h-index. However, with Journal of Marketing, Journal of Marketing Research and Journal of Consumer Research, Marketing Science is still one of the four journals with the highest h-index in Marketing.
- Journal of Marketings JIF has also increased substantially over recent years. After an average of 2.35 in 2001-2002, it was 2.85 on average in 2003-2004, which would pretty much locate it right on the regression line. It then jumped up to an average of 4.5 in 2005-2006. A more detailed analysis showed that this high recent JIF was mainly due to two highly-cited articles published in 2004 (which are in fact the two most highly-cited papers in the entire 2001-2005 period) that dealt with two key new issues in marketing: return on marketing and the service dominant logic. In October 2007 these two articles had a combined total of 200 citations, whilst the remaining articles in 2004 on average had about 10 citations. The recently released JIFs for 2007 show that Journal of Marketings JIF dropped from 4.83 to 3.75.
Industrial Marketing Management and Journal of Business Research are the two most important outliers at the other end of the spectrum with a relatively high Google Scholar h-index in comparison to their modest JIF. Both journals publish a relatively large number of papers (350 to 600 over five years) increasing their chances of reaching a high h-index. They are also cited quite often in journals that are not ISI indexed, such as European Journal of Marketing, Journal of Business to Business Marketing and Journal of Business & Industrial Marketing, which increases their Google Scholar h-index over their JIF and better reflects their overall impact on the field of marketing.
Organization Behaviour/Studies; HRM & Industrial Relations
At 0.633 (p < 0.001) this sub-discipline has the lowest correlation between the JIF and the GS h-index, but it is still high and very significant. Excluding the Human Resource Management and Journal of Organizational Behavior Management outliers (see below) raises this correlation to 0.685 (p < 0.001).
As Figure 9 (see the end of this chapter) shows several journals have a high JIF in relation to their Google Scholar h-index. The most striking cases are Human Resource Management and Journal of Organizational Behavior Management.
Human Resource Management
Harzing & Van der Wal (2008) already noted the discrepancy between Human Resource Managements JIF and h-index. Further investigation and an extensive email exchange with Thomson ISI representatives revealed that Thomsons search query for this journals JIF included a substantial number of homographs referring to Human Resource Management Review, Human Resource Management Journal as well as books with Human Resource Management in their title.
As a result the JIF for Human Resource Management was erroneously inflated. In fact at 0.64 the recently released 2007 JIF for Human Resource Management is very substantially lower than the 2002-2006 JIF average of 2.00.
Journal of Organizational Behavior Management
The JIF for Journal of Organizational Behavior Management has fluctuated wildly, standing at for instance 1.79 in 2003 and 0.11 in 2004. In 2004 there were only two citations to articles published in 2003 and 2002. In 2003 there were no less than 52 citations to articles published in 2002 and 2001. All of the 34 citations to JOBM in 2002 came from JOBM articles (i.e. a 100% self-citation on a journal level), whilst 11 of the 18 citations to articles published in 2001 came from within the journal. Further investigation showed that a very large proportion of the citations to papers in 2002 consisted of within-issue citations in a special issue. The third and fourth most cited of all papers in JOBM between 2001 and 2005 were published in this issue.
- For the first paper seven out of its eleven citations were in the same special issue (the other four being in later issues of the same journal).
- For the second paper four of its ten citations were in the same special issue (five of the remaining six were in later issues of the same journal).
- Two other papers in this special issue had four (three) of their four (three) citations within the same issue. Three more papers in this special issue each had two of their two citations in this special issue.
- The 8th paper in this issue had four of its four citations in one and the same later issue of JOBM. The ninth paper in this special issue had no citations.
- Coincidentally, the special issues topic was "The Search for the Identity of Organizational Behavior Management". Calling this special issue might well have been the best thing the editor ever did for the identity of the journal.
The journals relatively low Google Scholar h-index is due to Google Scholar's less than complete records for this journal. Google Scholar appears to have processed the papers published in the special issue, but does not seem to have parsed the citations in these papers. In this case, however, I would argue that this is probably for the best and even though Google Scholar is to some extent at fault, it provides a more realistic assessment of this journals low impact beyond a seemingly rather small academic circle.
On the other end of the spectrum we find journals that have a high h-index in comparison to their JIF. Journal of Human Resources publishes many policy oriented pieces dealing with the public sector that are heavily cited in working papers and policy papers as well as books not included in ISI. As a result the h-index better reflects its impact beyond academia.
The Journal of Business Ethics, Human Relations and International Journal of Human Resources Management, all published out of Europe, combine a relatively large number of published papers with citations that are evenly spread across papers, as well as a high number of citations in European journals not indexed in ISI, all three of which increase their Google Scholar h-index in comparison to their JIF. Hence the h-index provides a more accurate reflection of their relatively broad impact on the field.
Overall conclusions about sub-discipline analysis
Overall, we have seen that there is a very substantial agreement between the ISI JIF and the Google Scholar h-index for most sub disciplines. This means that for those sub-disciplines that have very limited ISI coverage (Finance & Accounting, Marketing and General Management & Strategy) the Google Scholar h-index could provide an excellent alternative for the 56-70% of journals not covered in ISI, especially given that for these disciplines the correlation between JIF and h-index for journals that did have ISI coverage ranged between .72 and .89. However, even for the other sub-disciplines the additional coverage provided by Google Scholar could be useful. Where the ISI JIF and the Google Scholar h-index diverged this was generally caused by one of two factors.
Factor 1: JIF's sensitivity to one-hit wonders
The sensitivity of the JIF to individual highly cited papers, which - especially for journals that publish relatively few papers artificially inflates the JIF in comparison to the h-index. Using mean scores produces distorted results if distributions are non-normal. In this respect, the h-index is a more robust measure. It is true that the h-index is influenced by the number of papers published and hence that journals that publish a lot of papers have a better chance to reach a high h-index. However, journals producing a larger number of highly cited papers have more impact on the field even if the average article in this journal is less highly cited than the average article in a journal that publishes fewer articles.
Journals such as Strategic Management Journal, the Academy of Management Journal, Organization Science and Management Science have higher h-indices than the Journal of Marketing and the Academy of Management Review (on average 55 versus on average 46), whereas they have much lower JIFs (on average 2.32 versus on average 3.95). However, as we have seen above, the high JIFs of the latter two journals are mostly caused by a very limited number (2-4) highly cited papers. Hence, I would argue that the h-index provides a more accurate picture of the generally very similar standing of these six journals.
Factor 2: Google Scholar's broader coverage
The key reason why some journals showed a relatively high h-index in comparison to their JIF lies in the broader coverage of Google Scholar. There are four main aspects to this broader coverage.
- Some journals have a strong policy impact and their articles are highly cited in policy documents or NBER working papers, neither of which are included in ISI.
- Articles in other journals, often published outside North America, are heavily cited in non North-American journals that are often not ISI indexed.
- Third, articles in some journals, especially in the area of computer science/information systems, are highly cited in conference papers, which tend to be the most important publication outlets in this field.
- Finally, some sub-disciplines such as Strategy and International Business generally have a low coverage in ISI. As a result the h-index better reflect the broad coverage of these journals.
Overall, I therefore argue that the Google Scholar h-index provides a more comprehensive measure of journal impact, in terms of both the number of journals and the number of citations covered.