{"id":926,"date":"2009-02-02T20:45:57","date_gmt":"2009-02-03T00:45:57","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/sj\/2009\/02\/02\/gormless-statistical-studies-and-social-analysis\/"},"modified":"2009-02-02T22:47:11","modified_gmt":"2009-02-03T02:47:11","slug":"gormless-statistical-studies-and-social-analysis","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/sj\/2009\/02\/02\/gormless-statistical-studies-and-social-analysis\/","title":{"rendered":"Gormless statistical studies and social analysis"},"content":{"rendered":"<p>I haven&#8217;t posted a rant for a while, so I though I&#8217;d share my frustration with a paper I recently read, filtered through a more distant conversation with\u00a0<a href=\"http:\/\/www.aaronsw.com\/weblog\/\">Aaron Swartz<\/a>.<\/p>\n<p>I am interested in the bounty of social exploration open to those of us alive today &#8212; from creating new geography-neutral organizations and governing bodies to actively living multiple separated and separately valued lives more explicitly than every before.\u00a0 But sociology papers I read, by humanities scholars, technophiles, or artistic lawyers, seem to be all form and no substance.<\/p>\n<p>Here is a <a href=\"http:\/\/www.time.com\/time\/health\/article\/0,8599,1739601,00.html\">popular<\/a> coffee-table study from Yee &amp; Bailenson at\u00a0Stanford&#8217;s <a href=\"http:\/\/vhil.stanford.edu\/\">Virtual Human Interaction Lab<\/a>\u00a0about online experiences influencing offline life.\u00a0\u00a0 Enough people I respect (I have seen references to it twice in the past 2 days, prompting this post) refer to it off-hand to support related conversation, as though its conclusions were surely true, that I read it today.\u00a0 And publication and the attention of a Time.com article certainly lend credence to the study!<\/p>\n<p>Yet on reviewing it, I find almost no aspects of it that are not misleading, naive,\u00a0or methodologically suspect.\u00a0\u00a0Group\u00a0selection seems lacking in depth, and the notion of &#8220;repeating&#8221; a study in the sense of verifying its results is used extremely loosely.\u00a0\u00a0The choice of motivation and introduction for the experiments are artificial and described in haste.\u00a0\u00a0Obvious dependence between different parts of an experiment are treated as unexpected discoveries (only after encountering results contrary to a hypothesis) rather than prepared for in advance.\u00a0 The language used obfuscates what is new about the study and what is universal, statistics are used as a bludgeon and to hide unconvincing results, and the interpretation of data\u00a0in terms of human psychology is\u00a0exaggerated in places and unsupported in others.\u00a0\u00a0\u00a0<\/p>\n<p>Here is <a href=\"http:\/\/vhil.stanford.edu\/pubs\/2008\/yee-proteus-implications.pdf\">the original paper<\/a>.<\/p>\n<p>This\u00a0sort of practice seems commonplace; I do not mean to single out the authors, who for all I know are models of methodology in their subfield.\u00a0 Doing a study as an excuse to posit a personal opinion can effectively produce centuries of dialectic without furthering our understanding of fundamentals one whit; which may be valuable in its own right and enough for <strong><a href=\"http:\/\/nymag.com\/arts\/books\/reviews\/39578\/\">certain people<\/a><\/strong>.\u00a0 But it is not good for exploring of the boundaries of what is possible, or for discovering deeper truths, and it sure doesn&#8217;t satisfy me.\u00a0\u00a0<\/p>\n<p><!--more--><\/p>\n<p>Specific failures of this and similar studies, to my lay eyes:<\/p>\n<ul>\n<li>Use of\u00a0citations to back up common sense statements at the beginning&#8230; and no citations to back up more dubious claims at the end.\u00a0 This implies a lack of prioritization of what needs measurement.\u00a0 It also\u00a0lends an aura of inevitability to the use of research to validate any statement.\u00a0\u00a0 &#8220;<span style=\"font-family: Times New Roman\">Research in communication and social psychology has shown that people automatically mimic each other\u2019s speech patterns and posture&#8221;<\/span><\/li>\n<li>Use of insufficiently detailed studies to describe a general phenomenon.\u00a0 Studying <span style=\"font-family: Times New Roman\">&#8220;whether wearing black uniforms causes athletes to behave more aggressively&#8221;\u00a0<\/span>takes up 2 pages of the background for the paper.\u00a0\u00a0Apparently the only corrollary studies have been about black v. white uniforms and nuns uniforms v. Ku Klux Klan robes.\u00a0 A delightfully unconvincing\u00a0spectrum of how the color or type of one&#8217;s clothes implicitly affects how one behaves.\u00a0 (Now if you took a dozen colors and patterns of uniform, tried to come up with some mapping of uniform style to statistical behavior on the field from one large set of data, and applied it with uniform predictive effect to a second large set of data, that might be interesting.)\u00a0<\/li>\n<li>Carrying out of experiments as &#8220;<em>find data that prove my idea<\/em>&#8221; exercises.\u00a0 When every hypothesis you make is supported by your data, and any\u00a0contrary evidence is described as suspect or indicative of a potential source of error, that&#8217;s a bad sign.\u00a0 <em>&#8220;these two unexpected findings show that a negotiation task performed in series does not provide a clean measure for the intended effects&#8221;<\/em> <strong>(oh no: unexpected findings!)<\/strong><\/li>\n<li>The use of dichotomies rather than nuanced metrics, particularly in interpreting ambiguous results to create an overall qualitative story when there is no significant quantitative one.\u00a0<br \/>\nHere: &#8220;tall v. short&#8221;, &#8220;aggressive v. passive&#8221;, &#8220;attractive v. ugly&#8221;<\/li>\n<li>The glossing over of potential sources of error, and total lack of any numerical error analysis.\u00a0 In more extreme cases, the glossing over of the existance of dependent variables, treating them as independent in some places (even while noting elsewhere that they do depend on one another).<br \/>\nThe point where I stopped reading this paper the first time through (I came back when posting this to read it completely) was where it explains that it was too difficult to assess the actual attractiveness of each avatar in WoW, so they used <em>race<\/em> a <strong>proxy<\/strong> for attractiveness.\u00a0 A few paragraphs earlier, the paper notes that each race has a uniform height.\u00a0 So the two major &#8216;independent&#8217; variables, height and attractiveness, are both <strong>determined uniquely<\/strong> by race &#8211; and there are only 8 races in the game, with very definite differences in traits, abilities, and characteristics.\u00a0 Such a paper could only have been published by an editorial board that had never played such games&#8230;<\/li>\n<li>The use of detailed measurement and statistics for values that are irrelevant to the main thrust of the study.\u00a0 This implies a lack of prioritization or recognition of where measurement matters.\u00a0\u00a0Here: measuring the height of each race by pixel, the use of 4 significant digits in reporting data.<\/li>\n<li>General lack of sanity checks.\u00a0 In the WoW experiment : level of a character was chosen to indicate how successful the character was, and by association how confident the player felt while playing it.\u00a0 however, players can have as many characters as they like.\u00a0 the first order effect of a statistically\u00a0attractive character is that the only person who ever sees the\u00a0char as more than\u00a0a flash on the screen &#8211; the player &#8211; is more likely to want to play it more often.\u00a0 hence, to play with that char for longer.\u00a0 So, since time spent playing the game rather than in real life is the main determinant of what level it reaches, you might expect that the maximally attractive chars will have high levels.\u00a0 Indeed, since every player can choose to optimize all character traits, while you can imagine players who really like playing very pretty or very homely characters, there may be fewer who want to play the most homely pretty-race character or the most-pretty homely-race character&#8230; a possible explanation for the inverse stat in the study that the characters of attractive short races tended to be lower level than those of uglier short races.\u00a0 (there, I can <em>pilpul<\/em> with the worst.)<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>I haven&#8217;t posted a rant for a while, so I though I&#8217;d share my frustration with a paper I recently read, filtered through a more distant conversation with\u00a0Aaron Swartz. I am interested in the bounty of social exploration open to those of us alive today &#8212; from creating new geography-neutral organizations and governing bodies to [&hellip;]<\/p>\n","protected":false},"author":1202,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-926","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7iVvB-eW","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/926","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/users\/1202"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/comments?post=926"}],"version-history":[{"count":0,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/926\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/media?parent=926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/categories?post=926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/tags?post=926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}