The Texas sharpshooter story
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
TRICKING PEOPLE INTO THINKING YOUR SCIENTIFIC SHOTS NEVER MISS
You visit the farm of a Texan, Joe, who claims to be a sharpshooter. When walking past his barn, you see a chalk target drawn on the wall with a bunch of tightly-grouped bullet holes in the bullseye. After observing that Joe can’t shoot well at all, you realize that he drew the bullseye after firing the shots.
The tale of the Texas sharpshooter resonates with JDM (Judgment and Decision-Making) research on perceiving illusory patterns, and a topic of recent interest, detecting bogus experimental results. In a recent paper, Ulrich Schimmack talks about multi-study research papers in this way. When you see 10 studies in a single paper that confirm a hypothesis, can you conclude that the basic effect is replicable and robust?
One problem in science is that reading a research article is a bit like visiting Joe’s farm. Readers only see the final result, without knowing how the final results were created. Is Joe a sharpshooter who drew a target and then fired 10 shots ar the target? Or was the target drawn after the fact?
-Schimmack, in press
We have been looking into the roots of the Texas Sharpshooter vignette in academic writing. The earliest and most common “initial” cite we found after a quick search was Grufferman’s from 1977, with no claims that this is the earliest use:
Here it is.
There have been several dramatic time-space clusters of leukemia reported in which, following an initial observation of two or more cases in a locality, a time unit and geographical area are selected so as to best define a time-space cluster. Such a posteriori clusters are analogous to the story of the Texas sharpshooter who would shoot his rifle at the side of a barn and then carefully draw a target around each bullet-hole so that each bullet-hole passed exactly through the center of the “bull’s-eye.” Although a posteriori clusters do serve to demonstrate that cases can cluster in time and space, they do not allow for determining whether this is more than a chance occurrence.
-Grufferman (1977)
REFERENCES
Grufferman S. (1977). Clustering and aggregation of exposures in Hodgkin’s disease. Cancer 39, 1829-1833
Schimmack, U. (in press). The Ironice Effect of Significant Results on the Credibility of Multiple Study Articles. Psychological Methods.
Photo credit: http://www.flickr.com/photos/24730945@N03/4130123404/sizes/l/
[…] Geschichte vom texanischen Scharfschützen umschreibt einen Aspekt, der dem file-drawer-Problem ähnelt. Wenn man eine wissenschaftliche […]
October 6, 2012 @ 1:46 pm