{"id":1417,"date":"2012-06-14T09:24:59","date_gmt":"2012-06-14T13:24:59","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/pamphlet\/?p=1417"},"modified":"2012-06-13T14:36:30","modified_gmt":"2012-06-13T18:36:30","slug":"more-reason-to-outlaw-impact-factors-from-personnel-discussions","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2012\/06\/14\/more-reason-to-outlaw-impact-factors-from-personnel-discussions\/","title":{"rendered":"More reason to outlaw Impact Factors from personnel discussions"},"content":{"rendered":"<p>Pity the poor,\u00a0beleaguered \u201c<a href=\"http:\/\/en.wikipedia.org\/wiki\/Impact_factor\">Impact Factor<\/a>\u2122\u201d (IF), a secret mathematistic formula originally intended to serve as a proxy for journal quality. <a href=\"http:\/\/cameronneylon.net\/blog\/warning-misusing-the-journal-impact-factor-can-damage-your-science\/\">No<\/a> one <a href=\"http:\/\/jcb.rupress.org\/content\/179\/6\/1091\">seems<\/a> to <a href=\"http:\/\/chronicle.com\/article\/The-Number-That-s-Devouring\/26481\">like<\/a> it much.\u00a0The manifold problems with IF have been rehearsed to death:<\/p>\n<ul>\n<li>The calculation isn&#8217;t a proper average.<\/li>\n<li>The calculation is statistically inappropriate.<\/li>\n<li>The calculation ignores most of the citation data.<\/li>\n<li>The calculated values aren&#8217;t reproducible.<\/li>\n<li>Citation rates, and hence Impact Factors, vary considerably across fields making cross-discipline comparison meaningless.<\/li>\n<li>Citation rates vary across languages. Ditto.<\/li>\n<li>IF varies over time, and varies differentially for different types of journals.<\/li>\n<li>IF is manipulable by publishers.<\/li>\n<\/ul>\n<p>The\u00a0<a href=\"http:\/\/www.iciam.org\/QAR\/\">study by the International Mathematical Union<\/a>\u00a0is especially trenchant on these matters, as is\u00a0<a href=\"http:\/\/bjoern.brembs.net\/comment-n499.html\">Bjorn Brembs&#8217; take<\/a>. I don&#8217;t want to pile on, just look at some new data that shows that IF has been getting even more problematic over time.<\/p>\n<p>One of the most egregious uses of IF\u00a0is in promotion and tenure discussions. It&#8217;s been understood for a long time that the Impact Factor, given its manifest problems as a method for ranking journals, is completely inappropriate for ranking articles. As the <a href=\"http:\/\/www.ease.org.uk\/sites\/default\/files\/ease_statement_ifs_final.pdf\">European Association of Science Editors has said<\/a><\/p>\n<p style=\"padding-left: 30px\">Therefore the European Association of Science Editors recommends that journal impact factors are used only \u2013 and cautiously \u2013 for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes either directly or as a surrogate.<\/p>\n<p>Even Thomson Reuters says<\/p>\n<p style=\"padding-left: 30px\">The impact factor should be used with informed peer review. In the case of academic evaluation for tenure it is sometimes inappropriate to use the impact of the source journal to estimate the expected frequency of a recently published article.<\/p>\n<p>&#8220;Sometimes inappropriate.&#8221; Snort.<\/p>\n<table width=\"200\" align=\"right\" bgcolor=\"#F7EFE5\">\n<tbody>\n<tr>\n<td align=\"center\"><a href=\"http:\/\/blogs.law.harvard.edu\/pamphlet\/files\/2012\/05\/where-the-best-papers-are.png\"><img decoding=\"async\" src=\"http:\/\/blogs.law.harvard.edu\/pamphlet\/files\/2012\/05\/where-the-best-papers-are-300x230.png\" alt=\"Graph from Lozano et al., 2012\" width=\"200\" \/><\/a><\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: center\"><span style=\"color: #999999\">&#8230;the money chart&#8230;<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Check out the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Money_shot\">money chart<\/a> from the recent paper &#8220;<a href=\"http:\/\/arxiv.org\/abs\/1205.4328\">The weakening relationship between the Impact Factor and papers\u2019 citations in the digital age<\/a>&#8221; by George A. Lozano,\u00a0Vincent Lariviere, and\u00a0Yves Gingras.<\/p>\n<p>They address the issue of whether the most highly cited papers tend to appear in the highest Impact Factor journals, and how that has changed over time. One of their analyses looked at the papers that fall in the top 5% for number of citations over a two-year period following publication, and depicts what percentage of these do <em>not<\/em>\u00a0appear in the top 5% of journals as ranked by Impact Factor. If Impact Factor were a perfect reflection of the future citation rate of the articles in the journal, this number should be zero.<\/p>\n<p>As it turns out, the percentage has been extremely high over the years. The majority of top papers fall into this group, indicating that restricting attention to top Impact Factor journals doesn&#8217;t nearly cover the best papers. This by itself is not too surprising, though it doesn&#8217;t bode well for IF.<\/p>\n<p>More interesting is the trajectory of the numbers. At one point, roughly up through World War II, the numbers were in the 70s and 80s. Three quarters of the top-cited papers were not in the top IF journals. After the war, a steady consolidation of journal brands, along with the invention of the formal Impact Factor in the 60s and its increased use, led to a steady decline in the percentage of top articles in non-top journals. Basically, a journal&#8217;s imprimatur \u2014 and its IF along with it \u2014 became a better and better indicator of the quality of the articles it published. (Better, but still not particularly good.)<\/p>\n<p>This process ended around 1990. As electronic distribution of individual articles took over for distribution of articles bundled within printed journal issues, it became less important which journal an article appeared in. Articles more and more lived and died by their own inherent quality rather than by the quality signal inherited from their publishing journal. The pattern in the graph is striking.<\/p>\n<p>The important ramification is that the Impact Factor of a journal is an increasingly poor metric of quality, especially at the top end. And it is likely to get even worse.\u00a0Electronic distribution of individual articles is only increasing, and as the Impact Factor signal decreases, there is less motivation to publish the best work in high IF journals, compounding the change.<\/p>\n<p>Meanwhile, computer and network technology has brought us to the point where we can develop and use metrics that serve as proxies for quality <a href=\"http:\/\/total-impact.org\/\">at the individual article level<\/a>. We don&#8217;t need to rely on journal-level metrics to evaluate articles.<\/p>\n<p>Given all this, promotion and tenure committees should proscribe consideration of journal-level metrics \u2014 including Impact Factor \u2014 in their deliberations. Instead, if they must use metrics,\u00a0they should <a href=\"http:\/\/altmetrics.org\/manifesto\/\">use article-level metrics only<\/a>, or better yet, <em>read the articles themselves.<\/em><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Pity the poor,\u00a0beleaguered \u201cImpact Factor\u2122\u201d (IF), a secret mathematistic formula originally intended to serve as a proxy for journal quality. No one seems to like it much.\u00a0The manifold problems with IF have been rehearsed to death: The calculation isn&#8217;t a proper average. The calculation is statistically inappropriate. The calculation ignores most of the citation data. [&hellip;]<\/p>\n","protected":false},"author":2110,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[68],"tags":[],"class_list":["post-1417","post","type-post","status-publish","format-standard","hentry","category-scholarly-communication"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5pLfN-mR","jetpack-related-posts":[{"id":1440,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2012\/06\/11\/editorial-board-members-what-to-ask-of-your-journal\/","url_meta":{"origin":1417,"position":0},"title":"Editorial board members: What to ask of your journal","author":"Stuart Shieber","date":"Monday, June 11, 2012","format":false,"excerpt":"...good behavior... Harvard made a big splash recently when my colleagues on\u00a0the\u00a0Faculty Advisory Council to the Harvard Library\u00a0distributed a Memorandum on Journal Pricing. One of the main problems with the memo, however,\u00a0is the relatively imprecise recommendations that it makes. It exhorts faculty to work with journals and scholarly societies on\u2026","rel":"","context":"In &quot;scholarly communication&quot;","block_context":{"text":"scholarly communication","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/scholarly-communication\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":693,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2011\/02\/01\/the-tetrahedron-test-case\/","url_meta":{"origin":1417,"position":1},"title":"The Tetrahedron test case","author":"Stuart Shieber","date":"Tuesday, February 1, 2011","format":false,"excerpt":"Phil Davis's recent post over at The Scholarly Kitchen on whether open access might save the academic world some money misses the point of the COPE initiative and Harvard's open-access fund (HOPE). Davis speculates that for the case of one set of journals that happened to be mentioned in my\u2026","rel":"","context":"In &quot;open access&quot;","block_context":{"text":"open access","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/scholarly-communication\/open-access\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2271,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2015\/05\/07\/in-support-of-behavioral-tests-of-intelligence\/","url_meta":{"origin":1417,"position":2},"title":"In support of behavioral tests of intelligence","author":"Stuart Shieber","date":"Thursday, May 7, 2015","format":false,"excerpt":"\u2026\u201cblockhead\u201d argument\u2026 \"Blockhead by Paul McCarthy @ Tate Modern\" image from flickr user Matt Hobbs. Used by permission. Alan Turing proposed what is the best known criterion for attributing intelligence, the capacity for thinking, to a computer. We call it the Turing Test, and it involves comparing the computer\u2019s verbal\u2026","rel":"","context":"In &quot;Alan Turing&quot;","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":159,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2009\/06\/18\/dont-ask-dont-tell-rights-retention-for-scholarly-articles\/","url_meta":{"origin":1417,"position":3},"title":"&#8220;Don&#8217;t ask, don&#8217;t tell&#8221; rights retention for scholarly articles","author":"Stuart Shieber","date":"Thursday, June 18, 2009","format":false,"excerpt":"A strange social contract has arisen in the scholarly publishing field, a kind of \"don't ask, don't tell\" approach to online distribution of articles by authors.\u00a0 Publishers officially forbid online distribution, authors do it anyway without telling the publishers, and publishers don't ask them to stop even though it violates\u2026","rel":"","context":"In &quot;Alan Turing&quot;","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1849,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2013\/10\/30\/the-affordable-care-acts-contradictory-free-market-stance\/","url_meta":{"origin":1417,"position":4},"title":"The Affordable Care Act&#8217;s contradictory free market stance","author":"Stuart Shieber","date":"Wednesday, October 30, 2013","format":false,"excerpt":"\u2026in the upper 90's\u2026 apparently from Health Care for America Now! via logarchism.com. The Patient Protection and Affordable Care Act (ACA) limits the \u201cmedical loss ratio\u201d (MLR) that an insurer can have \u2014 the percentage of collected medical premiums that must go to medical services for the insured. The minimum\u2026","rel":"","context":"In &quot;policy&quot;","block_context":{"text":"policy","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/policy\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":588,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2010\/09\/07\/for-publishers-using-pmc-to-kill-multiple-birds-with-one-stone\/","url_meta":{"origin":1417,"position":5},"title":"For publishers, using PMC to kill multiple birds with one stone","author":"Stuart Shieber","date":"Tuesday, September 7, 2010","format":false,"excerpt":"Here's a clever way for a journal to efficiently and cost-effectively provide open access to its articles (at least in the life sciences): Use PubMed Central as the journal's article repository. This expedient has all kinds of advantages: You have to allow for PMC distribution anyway, in fields where much\u2026","rel":"","context":"In &quot;open access&quot;","block_context":{"text":"open access","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/scholarly-communication\/open-access\/"},"img":{"alt_text":"PubMed Central logo","src":"https:\/\/i0.wp.com\/www.ncbi.nlm.nih.gov\/corehtml\/pmc\/pmcgifs\/pmclogo.gif?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/1417","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/users\/2110"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/comments?post=1417"}],"version-history":[{"count":18,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/1417\/revisions"}],"predecessor-version":[{"id":1478,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/1417\/revisions\/1478"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/media?parent=1417"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/categories?post=1417"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/tags?post=1417"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}