{"id":2218,"date":"2012-03-09T06:06:18","date_gmt":"2012-03-09T10:06:18","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/sj\/?p=2218"},"modified":"2012-03-09T06:06:18","modified_gmt":"2012-03-09T10:06:18","slug":"j-b-s-haldane-on-statistical-fraud","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/sj\/2012\/03\/09\/j-b-s-haldane-on-statistical-fraud\/","title":{"rendered":"J. B. S. Haldane on Statistical Fraud"},"content":{"rendered":"

From Haldane<\/a>‘s 1941 essay in <\/em>Eureka\u00a0#6 on “<\/em>The Faking of Genetical Results<\/a>“, reproduced here with appropriate corrections and hyperlinks.<\/em><\/p>\n

\nMY\u00a0FATHER<\/a>\u00a0published a number of papers on blood analysis. In the proofs of one of them the following sentence, or something very like it, occurred: “Unless the blood is very thoroughly faked, it will be found that duplicate determinations rarely agree.<\/em>” Every biochemist will sympathise with this opinion. I may add that the verb “to lake,” when applied to blood, means to break up the corpuscles so that it becomes transparent.<\/p>\n

In genetical work also, duplicates rarely agree unless they are faked. Thus I may mate two brother black mice, both sons of a black father and a white mother, with two white sisters, and one will beget 10 black and 15 white young; the other 15 black and 10 white. To the ingenuous biologist this appears to be a bad agreement. A mathematician will tell him that where the same ratio of black to white is expected in each family, so large a discrepancy (though how best to compare discrepancies is not obvious) will occur in about 26 percent of all cases. If the mathematician is a rigorist he will say the same thing a little more accurately in a great many more words.<\/p>\n

A biologist who has no mathematical knowledge, and, what is vastly more serious, no scientific honour, will be tempted to fake his results, and say that he got 12 black and 13 white in one family, and 13 black and 12 white in the other. The temptation is generally more subtle. In one of a number of families where equality is expected he gets 19 black and 6 white mice. It looks much more like a ratio of 3 black to 1 white. How is he to explain it? Wasn’t that the cage whose door once seemed to be insecurely fastened? Perhaps the female got out for a while or some other mouse got in. Anyway he had better reject the family. The total gives a better fit to expectation if he does so, by the way. Our poor friend has forgotten the binomial theorem. A study of the expansion of\u00a0(1+x\/2)25<\/sup><\/strong>\u00a0would have shown him that as bad a fit or worse would be obtained with a probability of 122753 x 2-23<\/sup>, or .0146<\/strong>. There is nothing at all surprising in getting one family as aberrant as this in a set of 20. But he is now on a slippery slope.<\/p>\n

He gets his Ph.D. \u00a0He wants a fellowship, and time is short. But he has been reading\u00a0Nature<\/em>\u00a0and noticed two letters*\u00a0to that journal of which I was joint author, in which I might appear to have hinted at faking by my genetical colleagues. Thoroughly alarmed, he goes to a venal mathematician. Cambridge is full of mathematicians who have been so corrupted by quantum mechanics that they use series which are clearly divergent<\/a>, and not even proved to be summable. Interrupting such a one in the midst of an orgy of Bhabha<\/a> and benzedrine, our villain asks for a treatise on faking.<\/p>\n

“I am trying to reconcile Milne<\/a>, Born<\/a> and Dirac<\/a>, not to mention some facts which don’t seem to agree with any of them, or with Eddington,” replies the debauchee, “and I feel discontinuous in every interval; but here goes.<\/p>\n

“I suppose you know the hypothesis you want to prove. It wouldn’t be a bad thing to grow a few mice or flies or parrots or cucumbers or whatever you’re supposed to be working on, to see if your hypothesis is anywhere near the facts. Suppose in a given series of families you expect to get four classes of hedgehogs or whatnot with frequencies\u00a0p<\/em>1<\/sub>,\u00a0p<\/em>2<\/sub>,\u00a0p<\/em>3<\/sub>,\u00a0p<\/em>4<\/sub>, and your total is\u00a0S<\/em><\/strong>, I shouldn’t advise you to say you got just\u00a0Sp<\/em>1<\/sub>,\u00a0Sp<\/em>2<\/sub>,\u00a0Sp<\/em>3<\/sub>\u00a0and\u00a0Sp<\/em>4<\/sub>, or even the nearest whole number. Here is what you’d better do. Say you got\u00a0A<\/em>1<\/sub>,\u00a0A<\/em>2<\/sub>,\u00a0A<\/em>3<\/sub>\u00a0and\u00a0A<\/em>4<\/sub>, and evaluate<\/p>\n

\"\\chi^2<\/p>\n

Your\u00a0\"\\chi^2\"\u00a0has three degrees of freedom. That is to say you can say you got\u00a0A<\/em>1<\/sub>\u00a0red,\u00a0A<\/em>2<\/sub>\u00a0green and\u00a0A<\/em>3<\/sub>\u00a0blue hedgehogs. But you will then have to say you got\u00a0S<\/em><\/strong>–A<\/em>1<\/sub>–A<\/em>2<\/sub>–A<\/em>3<\/sub>\u00a0purple ones. Hence the expected value of\u00a0\"\\chi^2\"\u00a0is 3, and its standard error is\u00a0\"\\sqrt{6}\"; so choose your\u00a0A<\/em>‘s so as to give a\u00a0\"\\chi^2\"\u00a0anywhere between 1 and 6. This is called faking of the first order. It isn’t really necessary. You might have\u00a0\"p_1,\u00a0\"p_2,\u00a0\"p_4\u00a0and\u00a0A<\/em>1<\/sub>=9,\u00a0A<\/em>2<\/sub>=A<\/em>3<\/sub>=3,\u00a0A<\/em>4<\/sub>=1<\/strong>. The probability of getting this is\u00a0\"(16!, which is only just under .04<\/strong>. \u00a0However, it looks better not to get the exact numbers expected, and if you do it on a population of hundreds or thousands you may be caught out.
\n“Your second order faking is the same sort of thing. Supposing your total is made up of\u00a0n<\/em><\/strong>\u00a0families, and you say the\u00a0r<\/em><\/strong>th consisted of\u00a0a<\/em>r<\/em>1<\/sub>,\u00a0a<\/em>r<\/em>2<\/sub>,\u00a0a<\/em>r<\/em>3<\/sub>,\u00a0a<\/em>r<\/em>4<\/sub>\u00a0members of the four classes,\u00a0s<\/em>r<\/em><\/sub><\/strong>\u00a0in all: you take<\/p>\n

\"((a_{r1}<\/p>\n

and sum for all values of\u00a0r<\/em><\/strong>. Your total ought to be somewhere near 3n<\/em><\/strong>. The standard error is\u00a0\"\\sqrt{6n}\", and it’s better to be too high than too low. A chap called Moewus<\/a> in Berlin who counted different types of algae<\/a> (or so he said), got such a magnificent agreement between observed and theoretical results, that if every member of the human race had repeated his work once a month for 1012<\/sup><\/strong>\u00a0years, they might expect as good a fit on one occasion (though not with great confidence). So Moewus certainly hadn’t done any second order faking. Of course I don’t suggest that he did any faking at all. He may have run into<\/a>\u00a0one of those theoretically possible miracles, like the monkey typing out the text of Hamlet by mere luck. But I shouldn’t have a miracle like that in your fellowship dissertation.<\/p>\n

“There is also third order faking. The 4n<\/em><\/strong>\u00a0different components of\u00a0\"\\chi^2\"\u00a0should be distributed round their mean in the proper way. That is to say, not merely their mean, but their mean square, cube and so on, should be near the expected values (but not too near). But I shouldn’t worry too much about the higher orders. The only examiner who is likely to spot that you haven’t done them is Haldane<\/a>, and he’ll probably be interned as a Red before you send your thesis in. Of course you might get R. A. Fisher<\/a>, which would be quite as bad. So if you are worried about it you’d better come back and see me later.”<\/div>\n

Man is an orderly animal. He finds it very hard to imitate the disorder of nature. In fact the situation is the exact opposite of what the reader of Paley<\/a>‘s Evidences<\/em>\u00a0might expect. But the problem is an interesting one, because it raises in a sharp and concrete way the question of what is meant by randomness, a question which, I believe, has not been fully worked out. The number of independent numerical criteria of randomness which can be applied increases with the number of observations, but much more slowly, perhaps as its logarithm. The criteria now in use have been developed to search for excessive irregularity, that is to say, unduly bad fit between observation and hypothesis. It does not follow that they are so well adapted to a search for an unduly good fit. Here, I believe, is a real problem for students of probability. Its solution might lead to a better set of axioms for that very far from rigorous but none the less fascinating branch of mathematics.<\/p>\n

* see U. Philip and J. B. S. Haldane (1939).\u00a0Nature<\/em>,\u00a0143<\/strong>, p. 334. \u00a0and
\n  Hans Gr\u00fcneberg and J. B. S. Haldane (1940).\u00a0Nature<\/em>,\u00a0145<\/strong>, p. 704.\n<\/div>\n

Two closing comments by T. W. K\u00f6rner<\/a>, who found Haldane’s essay worth reprinting in his brilliant textbook<\/a> on Fourier analysis: <\/p>\n

“The reluctance of the scientific community to accept the possibility of fraud is illustrated by the fact that Moewus was still cited in the literature (and even spoken of as a possible Nobel prize winner) until 1953. However, no one else ever succeeded in repeating his experiments…<\/p>\n

Unfortunately the statistical war against fraud is now over and the cheaters have won. The kind of tests proposed by Haldane depended on the fact that ‘higher order faking’ required a great deal of computational work. The invention and accessibility of the computer means that the computational work involved has ceased to be a problem for the dishonest scientist. In the physical and biological sciences the possibility that others will attempt to replicate experiments may act as a sufficient deterrent but in purely statistical subjects like sociology and experimental psychology the poblems raised by potential fraud have still to be faced<\/em>.”<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"

From Haldane‘s 1941 essay in Eureka\u00a0#6 on “The Faking of Genetical Results“, reproduced here with appropriate corrections and hyperlinks. MY\u00a0FATHER\u00a0published a number of papers on blood analysis. In the proofs of one of them the following sentence, or something very like it, occurred: “Unless the blood is very thoroughly faked, it will be found that […]<\/p>\n","protected":false},"author":1202,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[210,213,209,1],"tags":[5553,3826,60479,136,291,4497],"class_list":["post-2218","post","type-post","status-publish","format-standard","hentry","category-chain-gang","category-metrics","category-popular-demand","category-uncategorized","tag-analysis","tag-fraud","tag-haldane","tag-mathematics","tag-science","tag-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7iVvB-zM","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/2218","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/users\/1202"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/comments?post=2218"}],"version-history":[{"count":13,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/2218\/revisions"}],"predecessor-version":[{"id":2232,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/2218\/revisions\/2232"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/media?parent=2218"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/categories?post=2218"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/tags?post=2218"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}