{"id":3981,"date":"2016-10-20T12:51:02","date_gmt":"2016-10-20T16:51:02","guid":{"rendered":"http:\/\/blogs.harvard.edu\/sj\/?p=3981"},"modified":"2016-10-20T13:44:36","modified_gmt":"2016-10-20T17:44:36","slug":"psych-statistics-wars-new-methods-are-shattering-old-guard-assumptions","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/sj\/2016\/10\/20\/psych-statistics-wars-new-methods-are-shattering-old-guard-assumptions\/","title":{"rendered":"Psych statistics wars: new methods are shattering old-guard assumptions"},"content":{"rendered":"

Recently, statistician Andrew Gelman has been brilliantly breaking down the transformation of\u00a0psychology (and social psych in particular) through its adoption of\u00a0and\u00a0creative<\/em> use<\/em>\u00a0of statistical methods, leading to\u00a0an improved understanding of how statistics can be abused in any field, and of how empirical observations can be [unwittingly and unintentionally] flawed. This led to the concept of p-hacking<\/a> and other methodological fallacies<\/strong> which can be observed in careless uses of statistics throughout scientific and public analyses. And, as these new tools were used to better understand psychology and improve its methods, existing paradigms and accepted truths have been rapidly changed over the past 5 years. This shocks and anguishes researchers who are true believers\u00a0in”hypotheses vague enough to support any evidence thrown at them<\/em>“, and have built careers around work supporting those hypotheses.<\/p>\n

Here is Gelman’s\u00a0timeline of transformations in psychology\u00a0and in statistics<\/a>, from Paul Meehl’s argument in the 1960s that results in experimental psych may have no predictive power, to PubPeer<\/a>, Brian Nosek’s reprodicibility project<\/a>, and the current sense that “the emperor has no clothes”.<\/p>\n

Here is a beautiful discussion<\/a>\u00a0a week later, from Gelman, about how researchers respond to statistical errors or\u00a0other disproofs of part of their work. \u00a0In particular, how co-authors handle such new discoveries, either together or separately.<\/p>\n

At the end,\u00a0one of its examples turns up\u00a0a striking example of someone taking these sorts of discoveries and updates to their work seriously: Dana Carney<\/a>‘s public CV<\/a> includes inline notes next to each paper wherever significant methodological or statistical concerns were raised, or significant replications failed.<\/p>\n

Carney makes an appearance in his examples because of\u00a0her\u00a0most controversially popular research, with Cuddy an Yap, on power posing<\/strong>. \u00a0A non-obvious result (that holding certain open physical poses leads to feeling and acting more powerfully) became extremely popular in the popular media, and has generated a small following of dozens of related extensions and replication studies \u2014 which starting in 2015 started to be done with large samples and at high power, at which point the effects disappeared. \u00a0Interest within social psychology in the phenomenon, as an outlier of “a popular but possibly imaginary effect”, is so great that the journal\u00a0Comprehensive Results in Social Psychology<\/a><\/em>\u00a0has an entire issue devoted to power posing<\/a> coming out this Fall.
\nPerhaps motivated by Gelman’s blog post, perhaps by knowledge of the results\u00a0that will be coming out in this dedicated journal issue [which she suggests are negative], she put out a full two-page summary of her changing views<\/a> on her own work over time, from conceiving of the experiment, to running it with the funds and time available, to now deciding there was no meaningful effect. \u00a0My hat is off to her. \u00a0We need this sort of relationship to data, analysis, and\u00a0error to make sense of the world. But it is a pity that she had to publish such a letter\u00a0alone, and that her co-authors didn’t feel they could sign onto it.<\/p>\n

Update<\/strong>: Nosek also wrote a lovely paper in 2012 on Restructuring incentives to promote truth over publishability<\/a>\u00a0<\/em>[with input from\u00a0the estimable\u00a0Victoria Stodden] that describes\u00a0many points at which researchers have incentives to stop research and publish preliminary results as soon as they have something they could convince a journal to accept.<\/p>\n","protected":false},"excerpt":{"rendered":"

Recently, statistician Andrew Gelman has been brilliantly breaking down the transformation of\u00a0psychology (and social psych in particular) through its adoption of\u00a0and\u00a0creative use\u00a0of statistical methods, leading to\u00a0an improved understanding of how statistics can be abused in any field, and of how empirical observations can be [unwittingly and unintentionally] flawed. This led to the concept of p-hacking […]<\/p>\n","protected":false},"author":1202,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[206,210,60484,205,14968,6034,213],"tags":[],"class_list":["post-3981","post","type-post","status-publish","format-standard","hentry","category-a-la-mod","category-chain-gang","category-citation-needed","category-glory-glory-glory","category-knowledge","category-meta","category-metrics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7iVvB-12d","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3981","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/users\/1202"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/comments?post=3981"}],"version-history":[{"count":4,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3981\/revisions"}],"predecessor-version":[{"id":3985,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3981\/revisions\/3985"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/media?parent=3981"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/categories?post=3981"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/tags?post=3981"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}