Judgment and Decision
Making, vol. 1, no. 2, November 2006, pp. 159161.
"Decisions from experience" = sampling error + prospect theory:
Reconsidering Hertwig, Barron, Weber & Erev (2004)
Craig R. Fox^{1}
Anderson School of Management
and Department of Psychology
University of California at Los Angeles
,
Liat Hadar
Anderson School of Management
University of California at Los Angeles
Abstract
According to prospect theory, people overweight low probability events
and underweight high probability events. Several recent papers
(notably, Hertwig, Barron, Weber & Erev, 2004) have argued that
although this pattern holds for "descriptionbased" decisions, in
which people are explicitly provided with probability distributions
over potential outcomes, it is actually reversed in
"experiencebased" decisions, in which people must learn these
distributions through sampling. We reanalyze the data of Hertwig et al.
(2004) and present a replication to determine the extent to which their
phenomenon can be attributed to sampling error (a statistical rather
than psychological phenomenon) versus underestimation of rare events
(i.e., judgmental bias) versus actual underweighting of judged
probabilities. We find that the apparent reversal of prospect theory
in decisions from experience can be attributed almost entirely to
sampling error, and are consistent with prospect theory and the
twostage model of decision under uncertainty (Fox & Tversky, 1998).
Keywords: decisions from experience, uncertainty, Prospect
Theory, twostage model, probability learning, decision weights.
Several recent papers have contrasted "decisions from description"
with "decisions from experience." In the former, people are
explicitly provided probability distributions over potential outcomes;
in the latter, probabilities and outcomes are not provided but must be
learned by sampling from these distributions. Perhaps most notable is
an experiment presented by Hertwig, Barron, Weber and Erev (2004,
hereafter HBWE) that was also described in detail by Weber, Shafir and
Blais (2004) and Hertwig, Barron, Weber and Erev (2005). In this
experiment students either saw six decision problems (e.g., gain 4
points with probability .8; gain 0 otherwise; for a list of all lottery
pairs see Table 1) or sampled outcomes (with replacement) from
unlabeled buttons associated with these pairs of payoff distributions.
After sampling draws from each button as many times as they wished,
participants indicated which lottery they preferred to play once for
real money. The authors characterize their results as follows: "In
the case of decisions from description, people make choices as if they
overweight the probability of rare events, as described by prospect
theory....in the case of decisions from experience, in contrast,
people make choices as if they underweight the probability of rare
events." They conclude their abstract with a "call for two different
theories of risky choice." Although we are sympathetic to the
investigation of decisions from experience, we believe that the call
for two different theories of risky choice is premature as HBWE's
results are driven almost entirely by sampling error and are consistent
with prospect theory (hereafter PT, Kahneman & Tversky, 1979; Tversky
& Kahneman, 1992) when it is applied to the probability distributions
of outcomes that participants actually sampled.
Table 1: Percentage of choices consistent with prospect theory (PT) for
each decision problem in Hertwig et al. (2004). H = option with the
higher expected value; L = option with the lower expected value (the
first number is the prize amount in points, the second number is the
objective probability). Underlining indicates the choice predicted by
PT, assuming objective probabilities and median value and weighting
function parameters reported by Tversky & Kahneman (1992). Entries in
the last three columns indicates percentages of responses compatible
with PT for decisions from description, and for decisions from
experience assuming objective probabilities and probabilities
experienced by participants, respectively.
 Options  Percentage of Participants Satisfying PT Assuming: 
(r)23
   Baseline:   
Decision    Decisions from  "Objective"  Experienced 
problem  H  L  description  probabilities  probabilities 
1
 4, .8
 3, 1.0
 64
 12
 56

2
 4, .2
 3, .25
 64
 44
 76

3
 3, 1.0
 32, .1
 64
 28
 68

4
 3, 1.0
 4, .8
 72
 44
 52

5
 32, .1
 3, 1.0
 48
 20
 84

6
 32, .025
 3, .25
 64
 12
 76

 63
 27
 69

To motivate our approach we note that economists since Knight (1921)
have distinguished decisions under risk, in which objective
probabilities of outcomes are known by the decision maker, from
decisions under uncertainty, in which they are not. The
experiment of HBWE thus compared decisions under risk (where lotteries
were explicitly described) to decisions under uncertainty (where
lotteries were unlabeled buttons). In decisions under uncertainty
people's beliefs may differ from socalled "objective" probabilities
for a variety of reasons. First, experienced probabilities may differ
from objective probabilitiesas HBWE note, it follows from the
binomial distribution that people are more likely to undersample than
oversample lowprobability events, and this sampling error will be
more pronounced the lower the probability and the smaller the size of
the sample taken.^{2} Second, people
judge the probabilities that they have experienced incorrectly. Thus,
the question arises to what extent HBWE's finding of "underweighting"
can be attributed to: (1) sampling error (the difference between
socalled "objective" probabilities and probabilities of outcomes
that participants actually experienced); (2) judgment error (the
difference between experienced probabilities and judged probabilities);
and (3) probability weighting (the difference between judged
probabilities and their impact on choices). Breaking down the source
of "underweighting" is critical to the interpretation of HBWE's
results because sampling error is a statistical rather than
psychological phenomenon, judgment error is a bias in belief rather
than preference, and the weighting of these judged probabilities
provides the most apt comparison to prospect theory.^{3}
If the predominant source of HBWE's effect is sampling error, one would
expect most choices to accord with prospect theory weights applied to
the probabilities that participants actually observed. To test this
notion we obtained HBWE's raw data and tallied the participants in the
experience condition (unlabeled buttons) whose choices were consistent
with PT assuming "objective" probabilities (that were known only to
the experimenter) versus "experienced" probabilities (proportions of
events that participants observed).^{4} For each gamble pair we calculated the
percentage of participants who chose the higher PTvalued gamble
assuming median value and weightingfunction parameters reported by
Tversky and Kahneman (1992). The results (see Table 1) are striking
and consistent over all lottery pairs: although most participants
"violate" PT when the analysis is applied to "objective"
probabilities as presented in HBWE, most conform to the predictions of
PT when applied to probabilities that participants experienced, at a
similar rate to decisions from description.
In order to distinguish judgment error from probability weighting we
replicated the method of HBWE, adding an elicitation of judged
probabilities. We asked 46 students at Ben Gurion University to (1)
sample outcomes from each pair of unlabeled lotteries, (2) choose their
preferred lottery, and (3) recall the possible outcomes of each lottery
and then estimate their respective probabilities, counterbalancing the
order of judgment and choice tasks (a fuller account of this experiment
will appear elsewhere; details can be obtained from the authors). The
median correlation among respondents between judged and experienced
probabilities was .97 and the median mean absolute error was .06,
suggesting that participants were very accurate.^{5} When Tversky & Kahneman's (1992) median value and
weightingfunction parameters were applied to "objective"
probabilities, most choices (60%) violated PT as in HBWE; however,
when this analysis was applied to judged probabilities, most choices
(63%) conformed to PT. Thus, participants apparently weighted judged
probabilities in decisions from experience just as they tend to weight
chance probabilities in decisions from description, consistent with
Tversky & Fox's (1995) "twostage model" of decision under
uncertainty (see also Fox & Tversky, 1998; Wakker, 2004). We expect
that this model would have fit even better had predictions been based
on individually measured PT parameters, which are known to vary
somewhat between participants, rather than group medians (see Gonzalez
& Wu, 1999).
We note that HBWE do find some interesting patterns in their data.
Notably, participants with small incentives are content to make
decisions based on small samples of information, and more recently
sampled information may have greater impact on their decisions.
However, our internal analysis of HBWE's data and our replication show
that the socalled "underweighting" of low probability events in
decisions from experience is driven almost entirely by a tendency to
undersample lowprobability eventsa statistical property of the
binomial distributionand has almost nothing to do with
underestimation of observed probabilities or a tendency to underweight
these probabilities. Likelihood judgments correspond very closely with
experienced probabilities and choices are consistent with overweighting
low probabilities, as characterized by prospect theory. Thus, the call
for "two different theories of risky choice" seems premature, and
future research on decisions from experience might instead explore
models of search rules (what information do people seek), models that
terminate search (how much information do they seek), and models of
bias in likelihood judgment.
References
Fox, C. R., & Tversky, A. (1998). A beliefbased account of
decision under uncertainty. Management Science, 44,
879895.
Gonzalez, R. & Wu, G. (1999). On the shape of the
probability weighting function. Cognitive
Psychology, 38, 129166.
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004).
Decisions from experience and the effect of rare events in risky
choice. Psychological Science, 15, 534539.
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2005). The
role of information sampling in risky choice. Chapter in K.
Fiedler & P. Juslin (Eds.) Information Sampling as a Key
to Understanding Adaptive Cognition. New York: Cambridge
University Press.
Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of
Decision Under Risk. Econometrica, 47, 263 291.
Knight, F. (1921). Risk, Uncertainty, and Profit.
Boston, MA: HoughtonMifflin.
Tversky, A., & Fox, C. R. (1995). Weighting Risk and
Uncertainty. Psychological
Review, 102, 269283.
Tversky, A., & Kahneman, D. (1992). Advances in
Prospect Theory: Cumulative Representations of Uncertainty,
Journal of Risk and Uncertainty,
5, 297323.
Wakker, P. P. (2004). On the Composition of Risk
Preference and Belief. Psychological
Review, 111, 236241.
Weber, E. U., Shafir, S., & Blais, A. R. (2004). Predicting
risk sensitivity in humans and lower animals: Risk as variance or
coefficient of variation. Psychological Review, 111,
430445.
Wu, G. & Gonzalez, R. (1999). Nonlinear decision weights in choice
under uncertainty. Management Science, 45, 7485.
Footnotes:
^{1}This work was supported in part by the Fulbright Program and the
UCLABGU Program at the UCLA International Institute.
Address Correspondence to:
Craig R. Fox,
UCLA Anderson School,
110 Westwood Plaza #D511,
Los Angeles, CA 900951481,
Voice: 310/2063403, Email: craig.fox@anderson.ucla.edu
^{2}To illustrate, consider an
"experiencebased" choice between (a) a 2.5% chance of 32 points or
(b) a 25% chance of 3 points (Decision 6 in HBWE). If 100
participants sample outcomes from each lottery eight times
(approximately the median number reported in HBWE) one would expect 82
participants to sample exclusively zero outcomes for lottery (a) but
only 10 participants to sample exclusively zero outcomes for lottery
(b). Thus, one would expect 73 participants to face an apparent choice
between (a) receive nothing or (b) possibly receive 3 points. HBWE
characterize the choice of option (b) in this case as
"underweighting" of the lowprobability event
becauseunbeknownst to participants  the first lottery
has a higher expected value than the second lottery.
^{3}The
twostage model of decision under uncertainty holds that prospect
theory's inverseS shaped weighting function can be applied to judged
probabilities (Fox & Tversky, 1998). Strictly speaking, PT calls for
an alternative measure such as "bounded subadditivity" for decisions
under uncertainty (Tversky & Fox, 1995; see also Wu & Gonzalez,
1999). However, the data of HBWE do not lend themselves to such
alternative measures.
^{4}We thank Greg Barron for
providing us with these data.
^{5}Of course
the high correlation between judged and experienced probabilities may
be somewhat inflated by the wide range of "objective" probabilities
used in this experiment. However, if we examine the low probability
lotteries (p = .025  . 25) and high probability lotteries (p= .8  1)
separately, the correlations remain quite high: .84 and .98,
respectively.
File translated from
T_{E}X
by
T_{T}H,
version 3.74.
On 16 Nov 2006, 10:42.