If you go to a consumer research conference, you will hear tales of how experiments have undergone particular statistical rites: the attainment of the elusive crossover interaction, the demonstration of full mediation through Baron and Kenny’s sacred procedure, and so on. DSN has nothing against any of these ideas, but is opposed to subjecting all ideas to the same experimental designs, to the same tests, the same alternative hypotheses (typically a null of no difference), and the same rituals.

Zhao, Lynch, and Chen point out in their recent Journal of Consumer Research article that Baron & Kenny’s Mediation Analysis is incredibly popular (ca 13,000 cites between 1986 and 2010), prescribed reflexively, though flawed in ways its users probably aren’t aware of. This article was invited by the journal “to serve as a tutorial on the state of the art in mediation analysis”.

ABSTRACT
Baron and Kenny’s procedure for determining if an independent variable affects a dependent variable through some mediator is so well known that it is used by authors and requested by reviewers almost reflexively. Many research projects have been terminated early in a research program or later in the review process because the data did not conform to Baron and Kenny’s criteria, impeding theoretical development. While the technical literature has disputed some of Baron and Kenny’s tests, this literature has not diffused to practicing researchers. We present a nontechnical summary of the flaws in the Baron and Kenny logic, some of which have not been previously noted. We provide a decision tree and a step-by-step procedure for testing mediation, classifying its type, and interpreting the implications of findings for theory building and future research.

REFERENCES
Baron, Reuben M. and David A. Kenny (1986), Moderator-Mediator Variables Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations, Journal of Personality and Social Psychology, 51(6), 1173–82.

Bullock, J. G., Green, D. P, & Ha, S. E. (2010). Yes, But What’s the Mechanism? (Don’t Expect an Easy Answer), Journal of Personality and Social Psychology, Vol. 98, No. 4, 550–558.

Zhao, X., Lynch, J. G., Chen, Q. (2010).Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis. Journal of Consumer Research, 37, 197-206.

R Package for Causal Mediation Analysis

SPSS Code (see the Zhao, Lynch, and Chen article)

Who doesn’t like oxytocin? Who could dislike any substance referred to as a cuddle chemical? The answer may be you, if you are not in with the crowd feeling the effects of the hormone.

Carsten de Dreu and a super-long list of co-authors (listed below), have administered oxytocin to experimental participants and validated its bright side (cooperation among people in a group), but uncovered its dark side (defensive aggression towards people in other groups). Read all about it.

CITATION
Carsten K. W. De Dreu, Lindred L. Greer, Michel J. J. Handgraaf, Shaul Shalvi, Gerben A. Van Kleef, Matthijs Baas,Femke S. Ten Velden, Eric Van Dijk, Sander W. W. Feith. (2010) The Neuropeptide Oxytocin Regulates Parochial Altruism in Intergroup Conflict Among Humans. Science, 328(5984), 1408 – 1411.

ABSTRACT
Humans regulate intergroup conflict through parochial altruism; they self-sacrifice to contribute to in-group welfare and to aggress against competing out-groups. Parochial altruism has distinct survival functions, and the brain may have evolved to sustain and promote in-group cohesion and effectiveness and to ward off threatening out-groups. Here, we have linked oxytocin, a neuropeptide produced in the hypothalamus, to the regulation of intergroup conflict. In three experiments using double-blind placebo-controlled designs, male participants self-administered oxytocin or placebo and made decisions with financial consequences to themselves, their in-group, and a competing out-group. Results showed that oxytocin drives a “tend and defend” response in that it promoted in-group trust and cooperation, and defensive, but not offensive, aggression toward competing out-groups.

H/T author Michel Handgraaf
Photo credit 1: http://en.wikipedia.org/wiki/File:Oxytocin_with_labels.png
Photo credit 2: http://www.flickr.com/photos/markusschoepke/305865244/

John Lynch, Richard Netemeyer, Stephen Spiller, Alessandra Zammit have recently published in the Journal of Consumer Research this article on the propensity to plan and financial well being

ABSTRACT

Planning has pronounced effects on consumer behavior and intertemporal choice. We develop a six-item scale measuring individual differences in propensity to plan that can be adapted to different domains and used to compare planning across domains and time horizons. Adaptations tailored to planning time and money in the short run and long run each show strong evidence of reliability and validity. We find that propensity to plan is moderately domain-specific. Scale measures and actual planning measures show that for time, people plan much more for the short run than the long run; for money, short- and long-run planning differ less. Time and money adaptations of our scale exhibit sharp differences in nomological
correlates; short-run and long-run adaptations differ less. Domain-specific adaptations predict frequency of actual planning in their respective domains. A “very long-run” money adaptation predicts FICO credit scores; low planners thus face materially higher cost of credit.

And while reading the article is fun, it’s also a hoot to take the propensity to plan test yourself, and give it to your friends and family. Give it a whirl, see if it accords with their behavior. Here are the items. Feel free to post your score in the comments.

For each question, answer on a scale from 1 to 6 in which 1 means “I strongly disagree” and 6 means “I strongly agree.”
Propensity to Plan for Money—Short Run:
1. I set financial goals for the next few days for what I
want to achieve with my money.
2. I decide beforehand how my money will be used in
the next few days.
3. I actively consider the steps I need to take to stick to
my budget in the next few days.
4. I consult my budget to see how much money I have
left for the next few days.
5. I like to look to my budget for the next few days in
order to get a better view of my spending in the future.
6. It makes me feel better to have my finances planned
out in the next few days.

Propensity to Plan for Money—Long Run:
1. I set financial goals for the next 1–2 months for what
I want to achieve with my money.
2. I decide beforehand how my money will be used in
the next 1–2 months.
3. I actively consider the steps I need to take to stick to
my budget in the next 1–2 months.
4. I consult my budget to see how much money I have
left for the next 1–2 months.
5. I like to look to my budget for the next 1–2 months
in order to get a better view of my spending in the
future.
6. It makes me feel better to have my finances planned
out in the next 1–2 months.

Propensity to Plan for Time—Short Run:
1. I set goals for the next few days for what I want to
achieve with my time.
2. I decide beforehand how my time will be used in the
next few days.
3. I actively consider the steps I need to take to stick to
my time schedule the next few days.
4. I consult my planner to see how much time I have left
for the next few days.
5. I like to look to my planner for the next few days in
order to get a better view of using my time in the
future.
6. It makes me feel better to have my time planned out
in the next few days.

Propensity to Plan for Time—Long Run:
1. I set goals for the next 1–2 months for what I want
to achieve with my time.
2. I decide beforehand how my time will be used in the
next 1–2 months.
3. I actively consider the steps I need to take to stick to
my time schedule in the next 1–2 months.
4. I consult my planner to see how much time I have left
for the next 1–2 months.
5. I like to look to my planner for the next 1–2 months
in order to get a better view of using my time in the
future.
6. It makes me feel better to have my time planned out
in the next 1–2 months.

ARTICLE TEXT [Download]

MEDIA MENTIONS
Wall Street Journal: http://jcr.wisc.edu/publicity/authors/docs/SUNJ.AA.1A020.A1.361Z2009.pdf

Yahoo Finance: http://finance.yahoo.com/retirement/article/109540/fast-track-to-financial-success

Decision Science News (meta-reference): http://www.decisionsciencenews.com/2010/06/18/the-propensity-to-plan-is-good-for-your-wallet/

Blog posts about counterintuitive probability problems generate lots of opinions with a high probability.

Andrew Gelman and readers have been having a lot of fun with the following probability problem:

I have two children. One is a boy born on a Tuesday. What is the probability I have two boys? The first thing you think is “What has Tuesday got to do with it?” Well, it has everything to do with it.

DSN agrees with Andrew that one virtue of the “population-distribution” method is that it forces one to be explicit about various aspects of the problem, and in so doing, causes much confusion to disappear.

As a public service this week, Decision Science News presents the population-distribution representation of the problem (what it thinks of as the Gigerenzerian / Hoffragian / Peter Sedlmeier-ian representation of the problem) in a visual form.

To follow the logic, see Andrew’s post on how he solved the problem. Voila:

Red means “outside the reference class”. Yellow means “in the reference class but not boy-boy”. Green means “inside the reference class and boy-boy”.

Boy-boy in the reference class occurs with probability Green / (Green + Yellow) or 13 /27

NOTE
To see why DSN calls these Gigerenzerian / Hoffragian / Sedlmeierian representations, see:

Sedlmeier, P. (1997). BasicBayes: A tutor system for simple Bayesian inference.
Behavior Research Methods, Instruments & Computers, 29(3), 328-336.

Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102,, 684–704.

(Sorry for not using R, excel is just darn fast for some things)

Even people who aren’t avid baseball fans (your DSN editor included) can get something out of this one.

When two baseball teams play each other on two consecutive days, what is the probability that the winner of the first game will be the winner of the second game?

[If you like fun, write down your prediction.]

DSN’s father-in-law told him that recently the Mets beat the Phillies 9 to 1, but the very next day, the Phillies beat the Mets 10 to 0. How could this be? If the Mets were so good as to win by 8 points, how could the exact same players be so bad as to lose by 10 points to the same opponents 24 hours later?

Let’s call this situation (in which team A beats team B one one day, but team B beats team A the very next day) a “reversal”, and we’ll say the size of the reversal is the smaller of the two margins of victory. In the above example, the size of the reversal was 8.

Using R (code provided below), DSN obtained statistics on all major league baseball games played between 1970 and 2009 and calculated how often each type of reversal occurs per 100,000 pairs of consecutive games. The result is in the the graph above. Big reversals are rare. A reversal of size 8 occurs in only 174 of 100,000 games; a size 12 reversal happens but 10 times per 100k. A size 13 reversal never happened in those 40 years. One might think this is because it would be uncommon for a team that is so good to suddenly become so bad and vice versa, but note that big margins of victory are rare: only 4% of games have margins of victory of 8 points or larger.

Back to our question:

If a team wins on one day, what’s the probability they’ll win against the same opponent when they play the very next day?

We asked two colleagues knowledgeable in baseball and the mathematics of forecasting. The answers came in between 65% and 70%.

The true answer: 51.3%, a little better than a coin toss.

That’s right. When you win in baseball, there’s only a 51% chance you’ll win again in more or less identical circumstances. The careful reader might notice that the answer is visible in the already mentioned chart. The reversals of size 0, (meaning no reversal, meaning the same team won twice) occur 51,296 times per 100,000 pairs of consecutive games.

[At this point, DSN must admit that it is entirely possible that it has made a computational error. It welcomes others to reproduce the analysis with the code or pre-processed data at the end of this post.]

What of the adage “the best predictor of future performance is past performance”? It seems less true than Sting’s observation “History will teach us nothing“. Let’s continue the investigation.

Here were plot the probability of winning the second game based on obtaining various margins of victory in the first game. We simply calculated the average win rate for each margin of victory up to 11 games, which makes up 98% of the data, and bin together the remaining 2%, comprising margins of victory from 12 to 27 points. (Rest assured, the binning makes the graph look prettier, but does not affect the outcome.)

The equation of the robust regression line is: Probability(Win_Second_Game) = .498 + .004*First_Game_Margin which suggests that even if you win the first game by an obscene 20 points, your chance of winning the second game is only 57.8%

Still in disbelief? Here we do no binning and plot the margin of victory (or loss) of the first game winner as a function of its margin of victory in the first game. The clear heteroskedasticity is dealt with by iterative reweighted least squares in R’s rlm command. Similar results are obtained by fitting a loess line. This model is Expected_Second_Game_Margin = -.012 + .030*First_Game_Margin

One final note. The 51.3% chance you’ll win the second game given you’ve won the first is smaller than the so called “home team advantage”, which we found to be a win probability of 54.2% on first games and 53.8% on second games.

When the home team wins the first game, it wins the second game 54.7% of the time.
When the home team loses the first game, it wins the second game 52.8% of the time.
When the visitor wins the first game, it wins the second game 47.2% of the time.
When the visitor loses the first game, it wins the second game 45.3% of the time.

Surprisingly, when it comes to winning the second game, it’s better to be the home team who just lost than the visitor who just won. So much for drawing conclusions from winning. Decision Science News has always wondered why teams are so eager to fire their coaches after they lose a few big games. Don’t they realize that their desired state of having won those same few big games would have been mostly due to luck?

There you have it. Either we have made an egregious error in calculation or recent victories are surprisingly uninformative.

Do your own analysis alternative 1: The pre-processed data
If you wish, you can cheat and get the pre-processed data at http://www.dangoldstein.com/flash/bball/reversals.zip

This may be of interest for people who don’t use R or for impatient types who just want to cut to the chase.

No guarantee that our pre-processing is correct. It should be all pairs of consecutive games between the same two teams.

Do your own analysis alternative 2: The code

I’ll provide the column names file for your convenience at http://www.dangoldstein.com/flash/bball/cnames.txt. I left out a bunch of columns names I didn’t care about. The complete list is at: http://www.dangoldstein.com/flash/bball/glfields.txt

R CODE
(Don’t know R yet? Learn by watching: R Video Tutorial 1, R Video Tutorial 2)

#Home team advantage second game given home lost first game
#Equals 1 - Visitor p win second game given visitor won first game
with(relmerge[relmerge$dx < 0,],mean(dy > 0))


Decision Science News is happy with its purchase of Statistical Rules of Thumb by Gerald van Belle many years ago. It’s full of examples in which math can surprise.

The first example in the book is titled “use at least 12 observations in constructing a confidence interval”. When people first hear this they think, nonsense, there’s nothing magic about the number twelve.  And then they think that confidence interval sizes have to do with the square root of the sample size, but that still doesn’t do it. Thinking harder, one realizes that the half-width confidence interval for a sample of size n is t(n-1,1-alpha)/sqrt(n). One plots this out for 90% and 95% CIs and one sees that the first intuition was right, there is nothing magic about 12, but the plot above sure does seem to stop dropping in width somewhere around there. Maybe 15 is a safer number. To make it easier to see, here are the points on the above graph from the value 15 and greater.

We love heuristics for statistics, but do not promote following rules of thumb without reflection. We do promote playing with such rules of thumb as a way to become aware of the tradeoffs one makes in designing experiments. To encourage such play, we post the R code behind the above graphs here.

R CODE
(Don’t know R yet? Learn by watching: R Video Tutorial 1, R Video Tutorial 2)

n=seq(3,30,.1)
alpha=.1
y90=qt(1-alpha/2,n-1)/sqrt(n)
alpha=.05
y95=qt(1-alpha/2,n-1)/sqrt(n)

plot.new()
plot(n,y90,type=”l”,xlim=c(0,30),ylim=c(0,3),ylab=”Half-Width Confidence Interval Size”, xlab=”Sample Size”)
lines(n,y95,type=”l”)
text(15,y95[which(n==15)]+.15,labels=”95%”)
text(15,y90[which(n==15)]-.15,labels=”90%”)

#second plot
plot.new()
a=min(which(n>=15))
b=max(which(n>=15))
plot(n[a:b],y90[a:b],type=”l”,xlim=c(0,30),ylim=c(0,3),ylab=”Half-Width Confidence Interval Size”, xlab=”Sample Size”)
lines(n[a:b],y95[a:b],type=”l”)
text(15,y95[which(n==15)]+.15,labels=”95%”)
text(15,y90[which(n==15)]-.15,labels=”90%”)

Update: After Arjan’s comment, I tried to figure out if Van Belle is Dutch. I didn’t figure that out, but I did learn that he keeps a lot of these tips on his site. There’s even one on the 12 observation rule and some information added by others, including this figure:

Decision Science News knows that when faculty from London Business School travel abroad, they are frequently asked “how are things at the London School of Economics?” When the London Business School faculty members say politely that they are at LBS and not LSE, the askers suddenly look sympathetic, as if they’d inquired about a recently deceased pet.

It can be seen as a sensible reaction. The asker has heard of the London School of Economics, had a “false alarm” in thinking they recognized London Business School, and upon realizing that they have not heard of LBS, made the speedy inference it must not be very good. Perhaps this occurs through a variant of the recognition heuristic.

Decision Science News would like to point out that there is no need to feel sorry for London Business School faculty, who generally prefer being at a school of business over a school of economics, and further delight in the knowledge that London Business School was just rated the best MBA Programme in the world by the Financial Times.

Don’t believe their ranking? Well, one could consider the UK Government’s Research Assessment Exercise (RAE), according to which London Business School is the highest-scoring university in the UK for business, meaning it is higher ranked in business than Cambridge, Oxford, and yes, the London School of Economics with which it is so often confused.

Still unconvinced? Take a perspective from the USA, whose Forbes Magazine ranks LBS as the best MBA program outside the USA.

Need more data? See how the Economist credits LBS’s recent success: “What appears to be happening is that, as the job market for MBAs remains tough, more students are turning to schools with a worldwide reputation”.

So don’t cry for London Business School faculty, rest of world, congratulate them!

photo credit: http://www.forbes.com/

A few posts back, we showed how classic decision making experiments are being replicated on Amazon’s insta-subject-pool otherwise know as Mechanical Turk (aka MT).

After that, Steven Pinker, at the SJDM keynote, presented Mechanical-Turk-collected data on perceptions of whether the past or present is perceived as more violent.

This week, Decision News News, currently stationed at Yahoo! Research in New York, points to a useful guide to running experiments on MT, written by fellow Yahoo researcher and all-around wizard of computational psychology, Winter Mason.

If you want to do some very basic experiments using MT, you can probably get started using their templates. However, if you want to have participants engage in more complex interactive tasks, you probably want to use their command line tools or API. Here’s a guide to help you decide. If you go the command-line route, Winter’s instructions will save you some of the pain of figuring it out for yourself.

Here at Yahoo, we’ve been able to do some amazing MT experiments, including group decision making tasks, in which the groups are assembled on the fly. Think of it, no more inviting people to the lab and having to cancel when too few show up.

• Winter Mason’s Guide to doing experiments with Mechanical Turk
• Amazon’s Mechanical Turk Resource Center
• Amazon’s Mechanical Turk How To Guides
• Amazon’s Mechanical Turk Technical Documentation
• The resources page over at experimentalturk.wordpress.com

Photo credit: http://en.wikipedia.org/wiki/File:Tuerkischer_schachspieler_windisch4.jpg

Decision Science News has learned that the creator of expectancy-value theory, Martin Fishbein, has died.

From Icek Aizen:

We report with great sadness that our friend and colleague, Martin Fishbein, died Friday, November 27 of a heart attack while on a visit to London. Marty was a professor at the University of Illinois — Champaign/Urbana from 1961 to 1997 and since then has been the Harry C. Coles, Jr. Distinguished Professor of Communication at the Annenberg School for Communication and founding director of the Health Communication division of the Annenberg Public Policy Center.

Marty’s research interests included attitude theory and measurement, communication and persuasion, behavioral prediction and change, and behaviors in field and laboratory settings, including studies of the effectiveness of health-related behavior change interventions. He was president of both the Society for Consumer Psychology and the Interamerican Psychological Society and won many awards, including a Guggenheim Fellowship.

Marty is perhaps best known for his landmark theories in the field of social psychology. Working with his former student Icek Ajzen, he expanded his expectancy-value model into the theory of reasoned action, a theory that has had a marked impact not only on attitude research but also on applied work in such fields as health psychology, environmental behavior, marketing, organizational communication, and consumer behavior. His work is reviewed in a just published monograph: Fishbein, M., & Ajzen, I. (2010). Predicting and changing behavior: The reasoned action approach. New York: Psychology Press (Taylor & Francis).

Information about a celebration of his life will follow when plans for the event are finalized.

Icek Aizen
University of Massachusetts

For those wishing to learn more on Fishbein’s work see

• The Wikipedia article on Expectancy Value Theory
• The Wikipedia article on Fishbein and Ajzen’s Theory of Reasoned Action
• A free online edition of Fishbein & Azjen’s 1975 classic book Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research

We have all been there. You are running an experiment in which you would like participants to tell you what they believe. In particular, you’d like them to tell you what they believe to be the probability that an event will occur.

Normally, you would ask them. But come on, this is 2009. Are you going to leave yourself exposed to the slings and arrows of experimental economists? You need to give your participants an incentive to tell you what they really believe, right?

Enter the scoring rule. You pay off the subjects based on the accuracy of the probabilities they state. You do this by observing some outcome (let’s say “rain”) and you pay a lot of money to the people who assigned a high probability to it raining and you pay a little money (or even impose a fine upon) those who assigned a low probability to it raining. A so-called “proper” scoring rule is one in which people will do the best for themselves if they state what they truly believe to be the case.

Three popular proper scoring rules are the Spherical, Quadratic, and Logarithmic. Let’s see how they work.

Suppose in your experimental task you give people the title of a movie, and they have to guess what year the movie was released.  You tell them at the outset that the movie was released between 1980 and 1999: that’s 20 years. So you have these 20 categories (years) and you want people to assign a probability to each year. Afterwards, you will pay them out based on the actual year the movie was released and the probability they assigned to that year.

Let r be the vector of 20 probabilities, and r_1 could be the probability they assign to 1980 being the year of release, and r_2 the probability that it was 1981, so on through r_20 for 1999′s probability. Naturally, all the r’s add up to one, as probabilities like to do. Now, let r_i be the probability they assign to the year which turns out to be correct.

Under the Spherical scoring rule, their payout would be r_i / (r*r)^.5

Under the Quadratic scoring rule, the payout would be 2*r_i – r*r

Under the Logarithmic scoring rule, the payout would be ln(r_i)

In the movie above, the top row shows various sets of probabilities someone might assign to the 20 years. (Imagine the categories along the x-axis are the years 1980 to 1999).  Each bar in the graphs in the bottom three rows shows the person’s payout if that year turns out to be correct, based on the probabilities assigned to each year in the top row.

As you can see, when they assign a high probability to a category and it turns out to be correct, their payout is high. When they assign a low payout to a category and it turns out to be correct, their payout is low.

You’ll notice that the Logarithmic scoring rule goes right off the bottom of the page. This is because the log of small probabilities are negative numbers far beneath zero, and the log of 0 is negative infinity!

While I was at Stanford I heard that decision scientist extraordinaire Ron Howard (no relation) used to make students assign probabilities to the alternatives (A, B, C or D) on the multiple choice items on the final exam. The score for each question was the log of the probability they assigned to the correct answer. This means, of course, that if you assign a probability of 0 to alternative “B” and alternative “B” turns out to be correct, your score on that question is negative infinity. I always wondered if you got a negative infinity on one question if it meant you got negative infinity on the exam, or if there was some mercy clause.

But the main reason I am writing this post is because I wonder what experimental economists and psychologists are supposed to do when implementing log scoring rules in the lab. Naturally, you can endow the participant with cash at the beginning of the experiment and have them draw down with each question, but what do you do if they score a negative infinity? Take their life savings?

Winkler (1971) decided that he would treat probabilities less than .001 as .001 when it came time to imposing the penalty. Does anyone know of other methods?

REFERENCE

Robert L. Winkler (1971)  Probabilistic Prediction: Some Experimental Results, Journal of the American Statistical Association, Vol. 66, No. 336.  pp. 675-685.

NOTE

To make this simulation, I’ve drawn on the top row various beta distributions of differing modes between two fixed endpoints. This is akin to having a min and a max guess for the year of release, then entertaining various years between those two endpoints as most likely.