Vincent Larivière - Journal Impact Factors: history, limitations and adverse effects
Download as PPTX, PDF1 like1,298 views
The document discusses the history and implications of journal impact factors (JIF), highlighting its origins, critiques, and adverse effects on scientific communication and research evaluation. It outlines the evolution of citation metrics, concerns regarding their misuse, and the influence of these indicators on researchers' publication strategies. The authors question the necessity and effectiveness of JIF in serving the scientific community and evaluating research quality.
Vincent Larivière - Journal Impact Factors: history, limitations and adverse effects
- 1. Journal Impact Factors: history, limitations and adverse effects Vincent Larivière Cassidy R. Sugimoto [email protected] @lariviev crc.ebsi.umontreal.ca
- 2. What will this guy talk about? •Brief history of journal-level indicators •ISI and the Journal Citation Report (JCR) •Critiques of the JCR •Adverse effects •Why do we need journal level indicators?
- 3. Gross & Gross (1927) College Libraries and Chemical Education • Chemistry professors (US college) o “What files of scientific periodicals are needed in a college library to successfully prepare the student for advance work, taking into consideration also those materials necessary for the stimulation and intellectual development of the faculty?” • Citations provide an “objective” indicator o “One way to answer this question would be merely to sit down and compile a list of those journals which one considers indispensable. Such a procedure might prove eminently successful in certain cases, but it seems reasonably certain that often the result would be seasoned too much by the needs, likes and dislikes of the compiler.”
- 4. Gross & Gross (1927) • Compiled references made by 1926 papers published in the Journal of the American Chemical Society. o Excluded journal self-citations o Excluded cited papers from 1926 (to account for diffusion lag) • This resulted in 3,633 references to 247 journals.
- 5. Follow-up analyses • Several papers (≈20) scaled up the method devised by Gross & Gross • Growing datasets (i.e. manual work) • Mostly mono-disciplinary • All from the U.S. Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635-649.
- 6. Institute for Scientific Information • First large-scale citation index • Inspired on Shepard’s Citations—created in 1873 for legal research • Science Citation Index (1961) o NSF and NIH funded; cost of 400,000$ o Genetics Citation Index (subset) o Officially launched in 1964 • Sample of most cited journals (Bradford’s Law)
- 7. • Garfield and Sher (1963): exploration of the 1961 citation data (1.4 million references) • First mention of a Journal Impact Factor: o “One of the most interesting correlations is the “journal impact factor.” In the usual citation count methods, as, e.g., Gross and Gross (4), the importance of n journal is determined by the absolute number of citations to it.” o “The first step in obtaining a more meaningful measure of importance is to divide the number of times a journal is cited by the number of articles that journal has published.” Institute for Scientific Information (2)
- 8. Garfield gets scooped! • Martyn, J., Gilchrist, A. (1968), An Evaluation of British Scientific Journals. Aslib. • Based on the Science Citation Index! o “All citations to 1,842 British journals in the broad areas of science and technology which appeared in the 1965 Science Citation Index for the years 1964 and 1963.” • Data by discipline • Asymmetry between numerator and denominator o All citations / citable items • 2 year citation window
- 9. 1975 Journal Citation Report • First Journal Citation Reports (JCR) and reporting of the Journal Impact Factor • Based on more than 4.2 million references made in 1974 by 400,000 papers published in about 2,400 journals • Justifies citations as indicators of usage: o “The more frequently a journal’s articles are cited, the more the world’s scientific community implies that it finds the journal to be a carrier of useful information” • For librarians: decisions about subscriptions • For researchers: identifying publication venues, especially multidisciplinarity ones • Garfield did not advocate for using the JCR to identify elite journals.
- 10. Impact Factor • Average number of citations received by articles published in a journal two years after their publication • Impact Factor of a given journal in 2017 would be calculated as follows: Number of citations received in 2017 by documents published in the journal in 2015-2016 ______________________________________ Number of articles and reviews published in the journal in 2016-2017
- 11. Asymmetry numerator / denominator Number and proportion of citations received by articles, reviews, non-citable items, and unmatched citations, for four journals from the field of biochemistry and molecular biology, as well as Nature and Science, 2014-2015 papers and 2016 citations Article Review Non-Citable Items Unmatched Citations N % N % N % N % Cell 20,885 78.6% 3,068 11.5% 601 2.3% 2,016 7.6% 869 27.564 30.410 10.3% Nat. Chem. Biol. 3,263 77.4% 378 9.0% 217 5.1% 356 8.4% 268 13.586 15.066 10.9% PLOS Biol. 3,088 85.3% 6 0.2% 237 6.5% 290 8.0% 384 8.057 9.797 21.6% FASEB J. 3,650 74.6% 235 4.8% 203 4.2% 802 16.4% 881 4.410 5.498 24.7% Nature 55,380 78.6% 3,925 5.6% 5,067 7.2% 6,047 8.6% 1,784 33.243 40.140 20.7% Science 45,708 73.0% 4,886 7.8% 5,657 9.0% 6,340 10.1% 1,721 29.398 37.210 26.6% Journal N. Citable Items JCR Impact Factor Symmetric Impact Factor % Increase
- 12. Inclusion of journal self-citations Percentage of journal self-citations, by discipline, for citations received in 2016 by papers published in 2014-2015 0% 5% 10% 15% 20% 25% All disciplines Biomedical Research Clinical Medicine Chemistry Mathematics Psychology Biology Health Social Sciences Engineering and Technology Physics Earth and Space Professional Fields Humanities Arts Percentage ofjournal self-citations
- 13. Skewness of distributions Distribution of citations received by articles and reviews, for four journals from the field of biochemistry and molecular biology, 2014-2015 papers and 2016 citations 0 5 10 15 20 25 30 35 40 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Cell (JIF = 30.410) 0 5 10 15 20 25 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Nat. Chem. Biol. (JIF = 15.066) 0 5 10 15 20 25 30 35 40 45 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations PLOS Biol. (JIF = 9.797) 0 20 40 60 80 100 120 140 160 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations FASEB J. (JIF = 5.498)
- 14. Skewness of distributions (2) Distribution of the number of journals, by proportion of papers that obtained the JIF value, 2014-2015 papers and 2016 citations 0 100 200 300 400 500 600 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Numberofjournals Proportion ofpapers thatobtained the JIF value
- 15. Length of citation window Number of citations (left panel) and cumulative number of citations (right panel), by year following publication for papers published in 1985. 0 0.5 1 1.5 2 2.5 3 3.5 0 5 10 15 20 25 30 Citations Year following publication Number of citations 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 30 Citations Year following publication Cumulative citations 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 Citations Yea C 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 30 Citations Year following publication Cumulative citations 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 10 15 20 25 30 Citations Year following publication Cumulative % of citations Biomedical Research Psychology Physics Social Sciences
- 16. Disciplinary comparison Mean and maximum JIF of journals, by discipline, 2014-2015 papers and 2016 citations Discipline Mean JCR JIF Maximum JCR JIF Mean N. Ref. M Biology 1.683 22.81 48.99 Biomedical Research 3.526 46.6 48.94 Chemistry 2.768 47.93 46.37 Clinical Medicine 2.976 187.04 41.94 Earth and Space 2.173 30.73 53.71 Engineering and Technology 1.989 39.74 36.35 Health 1.647 17.69 39.08 Mathematics 1.017 9.44 26.56 Physics 2.699 37.85 36.57 Professional Fields 1.565 11.12 53.51 Psychology 2.050 19.95 54.56 Social Sciences 1.199 6.66 49.09
- 17. Disciplinary comparison (2) Mean JIF values as a function of A) Number of references and B) Age of cited references R² = 0.523 0 1 2 3 4 5 6 0 10 20 30 40 50 60 MeanImpactFactor Number of references JIF and cited references R² = 0.4721 0 1 2 3 4 5 6 0 10 20 30 40 MeanImpactFactor Age of references JIF and age of references A B
- 18. Declining predictive power Correlation between the JIF and the citations received by papers, 2 year following publication y = 0.0011x - 1.9418 r2 = 0.4666 y = 0.003x - 5.7759 r2 = 0.579 y = -0.0049x + 9.9863 r2 = 0.4788 0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 1900 1920 1940 1960 1980 2000 2020 Annualcoefficientofdetermination(r2)
- 19. Declining predictive power Declining share of top 1% most cited papers for elite journals
- 23. Impact Factor inflation Impact Factor by journal as a function of rank, for years 1997, 2007, and 2016 0.001 0.01 0.1 1 10 100 1000 1 10 100 1000 10000 100000 ImpactFactor(logscale) Rank of journal (log scale) 2016 2007 1997
- 24. Post JIF indicators • Five-year Impact Factor (Clarivate) • Eigenfactor (Clarivate) and SCImago Journal Rank (Scopus) • Source Normalized Impact per Paper—SNIP (Scopus) • CiteScore • None of those indicators solve the main misuse of the JIF: the application of a journal-level indicator to papers and researchers
- 25. Systemic effects • Orients researchers’ publication strategies o and research topics o Weakens non-indexed journals • Impact Factor engineering o Optimizing citation window o Numerator/denominator and front matter o Coercitive citations • Cash per publication policies o Science/Nature article 20 times mean academic salary (China) o Disciplinary inequalities… • Multiplication of citation indicators o Anyone can find an indicator where one looks better o Knock off indicators
- 26. Knock off Impact Factors
- 27. Knock off Impact Factors
- 28. Do we need the JIF? What problem does the JIF solve? Information filtering and ranking. Are there other mechanisms for this? What are the effects of the JIF? Journal indicators reify the role of journals—especially anglo-american. Is such hierarchy necessary? The community must ask: Is the use of the Journal Impact Factor good for science? … AND EVEN MORE IMPORTANTLY: Is research evaluation good for science?
- 29. Vincent Larivière Cassidy R. Sugimoto [email protected] @lariviev crc.ebsi.umontreal.ca Obrigado!
- 30. Garfield’s 1972 paper • Citations received during the last quarter of 1969 (Multiplied by four to obtain a year estimate) o First calculation of an Impact Factor (except M & G) • Top 152 journals account for 50% of all references • “It is apparent, even from the makeup of this partial listing, that a good multidisciplinary journal collection need contain no more than a few hundred titles.” • “This analysis gives good reason for concern about any increase in the number of scientific and technical journals.” • Citation Analysis as a Tool in Journal Evaluation. Journals can be ranked by frequency and impact of citations for science policy studies.