It’s the effect size
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
HOT COHEN’S D EFFECT SIZE VISUALIZER
Visualization by Kristoffer Magnusson (@RPsychologist)
Social science research puts the p-value on a pedestal. The p value, or probability of the data given the null hypothesis is true, is seen as the gateway to publication, giving authors an incentive to “p hack“, or use various tricks to get p-values down below .05. And they do this despite the lord loving the .06 as much as the .05. We have written on this before:
Cartoon by Decision Science News
One gripe with the p-value is that statistical significance is cheap. Most plausible hypotheses become statistically significant when the sample size is large enough. Among other things, statistical significance is a function of sample size. In the age of mTurk-scale data, attaining statistical significance is easier than ever. We have heard it said that that if you draw a line anywhere through the belly of the United States, you’ll find a significant different in height on opposite sides of the line because of the massive sample size. But it may be a puny difference.
Enter the concept of effect size. Effect size gives one a way to think about the magnitude of effects, not just the probability of the data given the null hypothesis (aka, the p-value). One popular measure of effect size, Cohen’s D, is discussed along with in the beautiful visualization pictured above. Learn more from the article It’s The Effect Size, Stupid, from which this post gets its name. And learn why you need lots of data to estimate effect sizes from our friends at Data Colada.
Agreed — especially in the era of large-scale data collection, this distinction seems really important. The recent facebook study, for example, finds effect sizes that are so small we don’t really have a good language for them.. maybe something like “itsy bitsy eeny weenie” would be appropriate for the d = .01 found. Of course, the n = 700,000 means that these will also be incredibly precise measurements (p < .001 for most of the estimates examined)..
August 26, 2014 @ 11:06 am