Judgment and Decision Making, vol. 4, no. 1, February 2009, pp. 20-33
Cognitive abilities and superior decision making under risk: A protocol analysis and process model evaluationEdward T. Cokely* |
Individual differences in cognitive abilities and skills can predict normatively superior and logically consistent judgments and decisions. The current experiment investigates the processes that mediate individual differences in risky choices. We assessed working memory span, numeracy, and cognitive impulsivity and conducted a protocol analysis to trace variations in conscious deliberative processes. People higher in cognitive abilities made more choices consistent with expected values; however, expected-value choices rarely resulted from expected-value calculations. Instead, the cognitive ability and choice relationship was mediated by the number of simple considerations made during decision making — e.g., transforming probabilities and considering the relative size of gains. Results imply that, even in simple lotteries, superior risky decisions associated with cognitive abilities and controlled cognition can reflect metacognitive dynamics and elaborative heuristic search processes, rather than normative calculations. Modes of cognitive control (e.g., dual process dynamics) and implications for process models of risky decision-making (e.g., priority heuristic) are discussed.
Keywords: Risky choice, intelligence, working memory, numeracy,
cognitive control, dual process theory, information search,
rationality, expected value, protocol analysis, priority heuristic.
Human decision-making is constrained by its bounded rationality and does not always follow normative prescriptions (Gigerenzer, Todd, & the ABC Research Group, 1999; Kahneman, 2003; Payne, Bettman, & Johnson, 1993; Simon 1990). Nevertheless, individual differences in cognitive abilities and skills predict normatively superior judgment and decision-making (Frederick, 2005; Peters & Levin, 2008; Peters, Vastfjall, Slovic, Mertz, Mazzocco, & Dickert, 2006; Stanovich & West, 1998; 2000; 2008). A variety of theories, such as dual-process theories, attribute the individual differences to deliberative processes (Baron, 1985; De Neys, 2006; Evans, 2008; Frederick, 2005; Kahneman, 2003; Kahneman & Frederick, 2007; Sloman, 1996; Stanovich & West, 1998; 2000); however, the link between decision processes and abilities is largely uninvestigated. What are the cognitive processes that give rise to the relationship between cognitive abilities and superior decision making under risk?
Previous research has examined individual differences in decision making under risk in lotteries with known probabilities. For low stakes lotteries normative expected utility processes are assumed to be approximated with calculations that multiply probabilities by potential gains/losses, i.e., expected-value calculations (Frederick, 2005; Payne, Samper, Bettman, & Luce, 2008). Frederick has demonstrated that expected-value choices are associated with scores on the cognitive reflection test, which is designed to measure one’s reliance on more consciously controlled processes rather than automatic first impressions (e.g., Stanovich and West’s, 2000, deliberative System 2 rather than intuitive System 1). The three-problem cognitive reflection test, which is known to correlate with other general cognitive ability measures, consists of mathematical problems for which an immediate intuitive response is incorrect. Frederick demonstrated that higher scoring individuals did not exhibit the clear non-normative risk asymmetry for gains and losses predicted by prospect theory (Kahneman & Tversky, 1979). When given a choice between a gain of $100 versus a 75% chance of a $200 gain, prospect theory predicted risk aversion and the selection of the certain $100.1 However, people with higher cognitive reflection scores more often selected options with the higher expected values (i.e., the probability multiplied by the potential risky gain — $150) as compared to lower scoring individuals.
There are several candidate mechanisms that may account for the link between cognitive abilities and superior decision making under risk. For example, one can make expected-value choices by performing expected-value calculations. Frederick (2005) suggests that computation of expected values may play a role, although he notes that it is not likely the only factor. More generally, Stanovich and West (2000) suggest that individual differences in normative judgments and decisions often arise from working memory capacity limitations on computation, implying that high ability individuals may make expected-value choices via expected-value calculations.2 Other research indicates that individual differences in risky decision making may also arise from variations in one’s general knowledge and understanding of probabilities — i.e., one’s numeracy (Peters & Levin, 2008; Peters et al., 2006). People high in numeracy, particularly the ability to comprehend and transform probabilities, are less affected by attribute framing because they can readily transform items such as 74% correct into 26% incorrect and translate percentages to frequencies and vice versa. Thus, numeracy may allow better risky choices as a result of a more accurate subjective sense of the size of gains and losses or other probability related trade-offs.
Theories describing the actual cognitive processes commonly used for decision making under risk tend to be imprecise (Brandstätter, Gigerenzer, & Hertwig, 2006; 2008; Johnson, et al. 2008; Payne et al., 1993; Payne & Braunstein, 1978; Selart et al., 2006). Risky choice models are typically as-if models, as in the case of prospect theory, which does not describe the exact cognitive operations of choice but holds only that people act as-if they evaluate losses with a steeper utility curve (Johnson et al., 2008). One exception to as-if modeling is the priority heuristic which is a parameter free choice-outcome and cognitive process model (Brandstätter et al., 2006). According to the priority heuristic, decisions between sure versus risky options are the result of considering simple reasons for a decision in a fixed order, until a stopping rule is met. First, people consider minimum gains. If the minimum gains differ by 1/10 or more of the maximum gain (1/10 of the maximum gain rounded to the closest prominent number) consideration stops and people choose the option with the higher minimum gain. If necessary, they consider a second reason, the probability of the minimum gain. If the probabilities of the two options differ by 1/10 or more of the probability scale, consideration stops and people choose the option with the higher probability minimum gain. If necessary, they will consider a third reason and choose the option with the higher maximum gain. A similar set of reasons and stopping rules occur for choices between losses.
The priority heuristic has accurately described majority choice-outcomes in several theoretically important datasets (but for critical reviews see Birnbaum, 2008; Hilbig, 2008; Johnson et al., 2008). Some evidence also supports the priority heuristic process model as latencies to choose between two options have been greater for choices that require three considerations compared to one consideration (Brandstätter et al., 2006). However, the priority heuristic is silent on the potential cognitive processes that may mediate the relationship between cognitive abilities and superior decision-making. Given that the priority heuristic is designed to predict potentially non-normative majority choices we hypothesized that it may predict many participants’ choices and choice processes, although it would be unlikely to predict behavior of high ability individuals.
Table 1: Example of discrete model predictions and predicted verbal reports for a sample choice.
Expected value Priority heuristic
Heuristics and simple considerations are common and often effective bases for judgment and choice (Gigerenzer et al., 1999; Payne, Bettman, Coupey, & Johnson, 1992; Payne et al., 1993; Tversky & Kahneman, 1974). We hypothesized that the relationship between cognitive abilities and decision-making under risk would not necessarily arise from expected-value calculations, but could result from simple considerations of reasons as in the priority heuristic, and simple transformations of probability information as in the research by Peters et al. (2006; Peters & Levin, 2008). Theory suggests that variation in superior decision making does not necessarily need to rely on the exact use of calculations based on normative models but can result from greater reflectiveness or thoroughness in decision making (Baron, 1985; 1990). Variation in risky choice performance has been linked to differences in duration and type of information search (Mann & Ball, 1994; Payne & Braunstein, 1978; Selart et al., 2006). Working memory measures are also known to predict strategic differences in elaboration during encoding (Bailey, Dunlosky, & Kane, 2008; Cokely, Kelley, & Gilchrist, 2006; Guida, Tardieu, & Nicolas, 2008; McNamara & Scott, 2001) and differences in the number of hypotheses generated during probability judgment (Dougherty & Hunter, 2003; Thomas, Dougherty, Sprenger, & Harbison, 2008). Therefore, we hypothesized that elaborative heuristic search — i.e., more thorough exploration and representation of the problem space — would often be positively related to superior risky decision making. To test the elaborative heuristic search hypothesis and to more precisely trace cognitive processes, we conducted a protocol analysis.
| Table 2: Coding system for protocol analysis including examples of each consideration, mean considerations per trial (and standard deviations), and total observed considerations. |
| Protocol codes | Example Considerations: | Mean considerations | Total observed | |
| ($125 or 30% of $900) | ||||
| 1. Minimum differences | $125 is more than nothing | .01 (.04) | 26 | |
| 2. Maximum differences | $900 is a lot more than $125 | .30 (.20) | 712 | |
| 3. Recode probability | 30% chance is 70% to gain nothing | .13 (.14) | 298 | |
| 4. Probability low | 30% just won’t happen | .41 (.18) | 950 | |
| 5. Probability high | 30% will probably happen | .16 (.08) | 383 | |
| 6. 10% of maximum | 10% of $900 is about 100 | .00 (.00) | 3 | |
| 7. Avoid risks | I always want a sure thing | .13 (.11) | 296 | |
| 8. Avoid losses | I never want to lose anything | .03 (.04) | 76 | |
| 9. Maximum money | I want the most money | .08 (.10) | 189 | |
| 10. Value low | $125 is nothing | .11 (.10) | 254 | |
| 11. Value high | $900 is a lot of money | .17 (.13) | 405 | |
| 12. Expected value | 30% of 900 is more than $125 | .08 (.19) | 198 | |
| 13. Other-ambiguous | A is better than B | .22 (.20) | 517 |
Our experiment was designed to examine individual differences in decision processes. Process tracing was performed with retrospective verbal reports (Ericsson & Simon, 1980) in which participants verbally reported the exact thoughts they remembered in the order in which they occurred, immediately following their choices. When people consciously and deliberately consider information, such as comparing minimum gains or transforming information into different probabilities, these processes should be observable in participants’ protocols (Evans, 2008; Sloman, 1996). Verbal reports have previously been effectively used in related studies of choice (Rettinger & Hastie, 2003; Payne 1976; but for potential limitations see De Neys & Glumicic, 2008; Reisen, Hoffrage, & Mast, 2008). To illustrate this methodology, both an expected-value calculation and the priority heuristic process predict that participants should consider distinct types of information when making their choices. Verbalization of an expected value or an attempt to estimate one (e.g., “75% of $200 is definitely more than $100”) would provide evidence of expected-value type processes. Similarly, the priority heuristic makes predictions about what information will and won’t be considered for different lotteries, and in what order (Brandstätter et al., 2006) (Table 1). These predictions allowed us to develop a coding system to quantify the types and amounts of considerations that were consistent and inconsistent with processing products predicted by the priority heuristic and expected-value calculations. Protocol analysis codes were also derived from previous research (Rettinger & Hastie, 2001; 2003) and a pilot study (Table 2).
We hypothesized that protocol analysis would reveal a positive relationship between expected-value type choices and elaborative heuristic search (Baron, 1985; Payne, 1976; Selart et al., 2006; Simon, 1990), operationalized as the total number of different types of simple considerations verbalized (excluding expected-value calculations and ambiguous codes), regardless of output order (Table 2).3 We also hypothesized that elaborative heuristic search would at least partially mediate the relationship between cognitive abilities and superior decisions.4 More elaborative and thorough search processes were expected to include variations in the number of considerations (e.g., consider maximum gains and probabilities versus considering only maximum gains) as well as explorations of different aspects of problems (e.g., interpret the large difference between potential gains as a potential loss). Such variations could help some participants avoid overlooking valuable information or oppose the influence of framing effects (Peters et al, 2006).
Eighty undergraduate students from introductory psychology courses at Florida State University participated in partial fulfillment of course requirements and were tested individually. Four cognitive reflection scores were excluded as participants had seen the test in another experiment. Four working memory scores and two verbal reports were lost because of equipment failure. Seven participants did not receive numeracy scores due to a procedural error.
Ability measures included: (1) the operation span — a working memory capacity task that partially mediates relationships predicted by traditional intelligence instruments (Turner & Engle, 1989); (2) the cognitive reflection test (CRT) which assesses differences in cognitive impulsivity (System 1) versus more deliberative thinking (System 2) (Frederick, 2005); (3) a numeracy scale measuring understanding of numerical probabilities (Lipkus, Samsa, & Rimer, 2001; see Peters and Levin, 2008 for the 11 item scale).
The stimuli included 40 choice problems with hypothetical gains/losses presented in US dollars. Each choice consisted of one certain option and one risky option, balanced such that expected value and priority heuristic models made unique predictions on exactly half of the trials. Expected-value ratios of lotteries were on average near the indifference point (M = 2.07, range = .15 to 5.3, relative to the certain option) a range in which the priority heuristic is expected to predict choices (for discussion see Brandstätter et al., 2006; 2008). Expected-value calculations predicted equal numbers of risky choices for gains and losses; priority-heuristic predictions were asymmetric favoring risky choices for losses, but not gains. Priority heuristic also predicted that 60% of choices would involve less search (i.e., a single consideration of the minimum possible gains/losses relative to 10% of the maximum gain/losses) while the other 40% of choices required the maximum number of considerations (i.e., all possible steps of the priority heuristic). Risky option probabilities ranged from 1%-80% (Appendix).
Participants were tested individually. Responses were recorded by a head-mounted microphone. Verbal report instructions and warm up think-aloud problems were provided by an experimenter seated behind the participant. The experiment began with the cognitive reflection task followed by an example lottery. Participants were told that the experiment involved 40 such choices, all of which were presented in the same randomized order. Choices were presented from the top to the bottom of the screen with the first option (e.g., “A. gain $50”) displayed for two seconds before the second option appeared (e.g., “B. 50% to gain $400”). Choices remained on the screen until the participant made a selection and was prompted for a retrospective report. Lastly, participants completed the working memory span and numeracy measures, and were debriefed.
Following Brandstätter et al. (2006), a model competition was conducted. This analysis assessed the frequency with which each model predicted majority choices, across all choices. Binomial analysis indicated that expected-value calculations predicted majority choices significantly better than chance (M = .83, p = .001). A non-parametric test of equal proportions indicated that expected value also significantly outperformed the priority heuristic, χ2 = 12.17, p = .001, d = 1.3, which predicted at chance levels (M = .45, p > .5). A variety of subsequent analyses of the priority heuristic converged to suggest that in the current task environment the priority heuristic was an inaccurate process and choice-outcome model (see also Birnbaum, 2008; Hilbig, 2008; Johnson et al., 2008).5
Because abilities are known to influence choice, and given evidence on the limits of majority choice aggregation analyses (Regenwetter, Grofman, Popova, Messner, Davis-Stober, & Cavagnaro, 2008), we examined individual model-prediction-accuracy scores. Subsequent analyses compared the proportion of expected value consistent choices averaged across all choices for each individual. A one sample t test indicated that expected-value calculations strongly predicted participant choices (M = .72, SD = .12) above chance levels, t (79) = 16.02, p = .001, d = 1.9. The proportion of choices consistent with expected value was significantly related to CRT, r (74) = .27, p = .02, and numeracy, r (71) = .28, p = .02 (Table 6). A mixed model analysis of variance (ANOVA) with risk type (certain, risky) by choice type (gain, loss) by working memory span quartile (low, high) also indicated that working memory was associated with differences in choices, F (1, 36) = 7.70, p = .01, d = .8. High working memory span participants made significantly more expected-value type choices (M = .79, SD = .13) as compared to low span participants (M = .70, SD = .10).
Verbal reports were analyzed by two raters blind to model and judgment performance (Table 2). A randomly-selected subset of verbal reports (13%) were scored by both raters and indicated high inter-rater agreement on the number of considerations, r (8) = .97, p = .01, and substantial agreement on specific consideration codes (kappa = .63). The total number of considerations verbalized was also related to the mean choice reaction time, r (67) = .46, p = .001,6 indicating that individuals who retrospectively verbalized more considerations also took longer to make their judgments. Unless otherwise noted, seven participants were excluded from subsequent analysis because more than 50% of their verbal protocols were unrevealing (e.g., “A is better; I like B”).7
Figure 1: A linear regression with elaborative heuristic search (i.e., the number of verbalized considerations) predicting each participant’s overall proportion of expected-value choices (ambiguous and expected-value verbalizations are not included). The line is based on the regression.
Three individuals verbalized expected-value calculations (or estimations) nearly exclusively (95–100% of all trials; see Table 3 for examples). The frequency of verbalized expected-value calculations was significantly related to expected-value type choices, r (69) = .25, p = .03; however, expected-value calculations were unrelated to cognitive ability variables (Table 6). The remaining participants exhibited a clear relationship between the number of considerations verbalized and expected-value choices excluding any ambiguous or expected-value verbalizations, β = .60, t = 15.90, p = .001, R2 = .36 (Figure 1).8 Individuals who made the most expected-value choices (top quartile) verbalized about twice as many considerations per trial (M = 1.78, SD = .52) as did those who made fewer (bottom quartile) expected-value choices (M = .94, SD = .35). Across all participants, the number of considerations verbalized was also significantly related to CRT, r (72) = .23, p = .05; numeracy, r (69) = .36, p = .01; and working memory span, r (72) = .25, p = .04 (see Tables 4 and 5 for examples of verbal protocols; see Tables 6 and 8 for intercorrelations among variables).9 For example, across 40 trials, excluding expected-value or ambiguous verbalizations, individuals with higher working memory span scores (top quartile) verbalized significantly more considerations (M = 60.8, SD = 20.8) as compared to those with lower (bottom quartile) working memory scores (M = 47.1, SD = 14.5), F (1, 30) = 4.51, p = .04, d = .08.
Table 3: A sample of protocol analysis revealing expected-value calculation or estimation.
Choice options Protocol analysis sample: Expected-value verbalizations Loss: $50 or 5% chance to lose $4000 5% times $4000 is $200 certain loss or is it $20. No $200 which is more than $40 certain loss. Gain: $275 or 20% chance to win $900 10% chance of 900 is $90 which is $180. $180 or $275, $275 is more so yeah. 275 certain gain is more than 20% of 900 which is $180 gain. Loss: $120 or 5% chance to lose $1600 uhh crap 5% chance- .05 times 1600 is pretty sure $30 or $300. uhh its $30 so $120 is more than $300, no, whatever, so $300 is more. Gain: $80 or 3% chance to gain $5600 My first thought, 3% of 5600 would be more than A Loss: $275 or 20% chance to lose $900 20% okay 50% of 900 is 450, 900 times .20 is $180. $275 certain loss is more than $180 certain loss. Gain: $150 or 30% chance to gain $1080 Yeah its b. $150 is not or 30% of 1080 is more than $100 Loss: $200 or 1% chance to lose $3000 10% of 3000 is 300. 200 certain loss is way more than 1% of $3000 and that’s how I came about that. Loss: $50 or 50% chance to lose $400 uhh that’s easy 50% of- $200 is more than $50 certain loss. Gain: $50 or 5% chance to win $4000 My first thought, 5% of 4000 is more than B.
Table 4: A sample of coded protocol analysis from individuals with lower working memory, numeracy, and/or cognitive reflection scores (i.e., bottom quartile).
Choice Options Protocol Analysis Sample: Lower Ability Verbalizations Loss: $50 or 5% chance to lose $4000 5 percent isn’t that big of a percentile (probability low). Gain: $275 or 20% chance to win $900 A [risky choice], because it’s more money (max difference). Loss: $120 or 5% chance to lose $1600 1600 is a lot more than 120 so A [120] (max difference). Gain: $80 or 3% chance to gain $5600 First thought was wow, only 3%? Lame. (probability low). Loss: $275 or 20% chance to lose $900 My first thought was that 275 certain loss was less so I chose that one (max difference). Gain: $150 or 30% chance to gain $1080 Um, my first thought was to look at the percents and 30% of a $1080 gain is — those aren’t great chances (probability low) so I decided to pick the certain amount of money. Loss: $200 or 1% chance to lose $3000 B is a lot more riskier than A so I chose A (avoid risks). Loss: $50 or 50% chance to lose $400 50 percent is a lot (probability high). Gain: $50 or 5% chance to win $4000 My first thought was the percent is really low (probability low) so I went with the sure gain.
Table 5: A sample of coded protocol analysis from individuals with higher working memory, numeracy, and/or cognitive reflection scores (i.e., top quartile).
Choice Options Protocol Analysis Sample: Higher Ability Verbalizations Loss: $50 or 5% chance to lose $4000 My first thought was that 4000 is a lot to lose (value high) even though it’s only 5% chance (probability low) and only losing 50 compared to that (max difference) is not very bad at all. Gain: $275 or 20% chance to win $900 My first thought was that 20% chance is not likely to happen (probability low) and it was only 900 compared to B (max difference) which was 275 certain, so that’s why I chose B. Loss: $120 or 5% chance to lose $1600 My first thought was it’s only 5% chance (probability low) to lose 1600 and I have a 95% chance (recode probability) of not having lost anything. Gain: $80 or 3% chance to gain $5600 My first thought was I saw the 5600 dollars and I saw the 80 dollars but the 5600 dollars — there is still a chance for me to gain, it was really small (probability low), but you never know. That’s why I chose B because 5600 dollars is a lot of money (value high). Loss: $275 or 20% chance to lose $900 My first thought was with 20% chance of owing 900 dollars, that gives me 80% chance (recode probability) to not owe 900 dollars and I have pretty good chances (probability high). Gain: $150 or 30% chance to gain $1080 I chose the 30% chance of getting $1080 over certain chance er, certain that you’re getting 150. 150 is not a whole much (value low), you know? That’s not a whole much a lot of money and 1080 is a good amount bigger than 150 (max difference) even though there is only, I think there was only 30% chance of getting it (probability low). Loss: $200 or 1% chance to lose $3000 I chose the 1% of $3000 because that’s really small (probability low), it’s 1 in 100 (recode probability) of you actually losing $3000 compared to certain losing whatever—300. Loss: $50 or 50% chance to lose $400 I chose the $50 certain loss because it’s not a whole lot of money (value low), compare that to, I think it was, 50% chance of losing $400 so that’s a pretty big difference (max difference). Gain: $50 or 5% chance to win $4000 Uh I took the 5% chance of getting 4000 compared to 50 ‘cause 50 is really, really small compared to 4000 (max difference) and you have a 5% chance which is pretty small (probability low) but, uh, if you actually do gain that you gain a lot more than if you take 50.
We next constructed a series of hierarchical linear regression models (Table 7) with the most complex (full) model using three predictors including (1) expected-value verbalizations; (2) all three ability measures; and (3) number of verbalized considerations. The full model was a strong predictor of expected-value choices, F (5, 53) = 22.23, p = .001, R2 = .44. The number of verbalized considerations accounted for a moderate amount of unique variance, F (1, 53) = 8.15, p = .001, R2change = .24. The number of verbalized considerations also fully mediated the relationships between all three cognitive ability measures and expected-value choices (ts < 1, see Table 7).
Table 6: Intercorellations for main variables.
1 2 3 4 5 1. Expected-value choices . . . . . 2. Cognitive reflection test . . . . . 3. Numeracy . . . . . 4. Working memory . . . . . 5. Expected-value verbalizations . . . . . 6. Elaborative heuristic search . . . . . Notes: * p < .05; ** p < .01
Table 7: Hierarchical linear regression analysis explaining expected-value choices.
Models and variables β R R2 ΔR2 F Model 1. Expected-value calculations . . . . . Model 2. Ability variables added Expected-value calculations . . . . . Working memory span . . . . . CRT . . . . . Numeracy . . . . . Model 3. Number of considerations added Expected-value calculations . . . . . Working memory span . . . . . CRT . . . . . Numeracy . . . . . Elaborative heuristic search . . . . . Note: * p < .05; ** p < .01
To what extent might these results reflect the influence of particular choices, such as choices on gains rather than losses or choices involving high versus lower monetary values? To assess independent relationships controlling for these potentially influential factors we conducted a multilevel analysis. First, we constructed independent regression equations for each participant, predicting each participant’s responses across all 40 choice trails. Individual level regression equation coefficients (i.e., unstandardized β coefficients) were computed for each of the following variables (1) expected-value model choice predictions; (2) priority heuristic model choice predictions; (3) gain versus loss trails (to assess and control for potential asymmetries in responding); and (4) the highest absolute monetary value for each choice (to assess and control for potentially non-uniform influences of declining marginal utility).10 Next, we examined the intercorrelations between the individual level regression coefficients, all cognitive abilities, and the number of verbalized considerations (Table 8).11
As expected, results revealed reliable relationships between the expected-value choice coefficients and all cognitive ability measures including the cognitive reflection test, r (66) = .29, p = .02; numeracy, r (62) = .29, p = .02; and working memory span, r (66) = .27, p = .03. The number of verbalized considerations was also significantly related to the expected-value choice factor, r (69) = .45, p = .001. Lastly, a hierarchical linear regression was constructed, following the previous analyses but predicting the expected-value individual level coefficients with (1) expected-value verbalizations, (2) all three ability measures, and (3) number of verbalized considerations. The full model was again a strong and significant predictor, F (5, 53) = 4.22, p = .003, R2 = .29. The number of verbalized considerations also accounted for unique variance beyond other variables, F (1, 53) = 10.06, p = .003, R2change = .14, and again fully mediated the influence of all three cognitive abilities (ts < 1).
Table 8: Intercorrelations of ability, elaborative heuristic search, and individual level regression coefficients (indicated by β).
1 2 3 4 5 6 7 1. Expected value β . . . . . . . 2. Cognitive reflection test . . . . . . . 3. Numeracy . . . . . . . 4. Working memory . . . . . . . 5. Elaborative heuristic search . . . . . . . 6. Gains vs. losses β . . . . . . . 7. Maximum lottery value β . . . . . . . 8. Priority heuristic β . . . . . . . Note: * p < .05; ** p < .01
A very small minority of our sample (about 5%) consistently verbalized expected-value processes during decision making (Payne & Braunstein, 1978). The vast majority of expected-value choices were instead associated with simple heuristic-type decision processes. These decision processes were similar to the component considerations in the priority heuristic (see Table 2), although the priority heuristic was otherwise an inaccurate process and choice-outcome model. Consistent with the elaborative heuristic search hypothesis we found a relationship between the number of considerations verbalized and expected-value choices. Elaborative heuristic search also mediated the relationships between cognitive abilities and expected-value choices.12 These results demonstrate that neither deliberative thinking nor cognitive abilities are necessarily associated with normative calculations, even when associated with normatively superior decision performance.
The elaborative heuristic search captured by protocol analysis in the current experiment may, in part, result from differences in top-down, early selection cognitive control mechanisms used during the task (Jacoby, Kelley, & McElree, 1999). The prevailing theoretical framework emphasizes a late correction cognitive control interpretation of dual process dynamics. That is, when controlled processes (System 2) do not compute an answer they are assumed to primarily operate by monitoring and correcting the output of automatic processes (Kahneman, 2003). In contrast, early selection cognitive control uses controlled processing (System 2) to generate goals, strategies, and mental contexts that qualitatively alter the output of automatic processes (System 1) before critical impressions are yielded (Jacoby, Shimizu, Daniels, & Rhodes, 2005). For example, if some participants approached the task with the mindset of playing a game (e.g., “I feel lucky”) they would likely generate different search processes as compared to those construing choices in terms of their actual spending power (e.g., “the probability is low but I don’t even have $7000 dollars”). Spending-power type considerations (i.e., considering values small or large) were found to be significant predictors of expected-value choices [r (76) = .41, r (76) = .36, respectively] and were also strongly related to the overall number of considerations [r (76) = .62, and r (76) = .68, respectively]. Moreover, related research indicates that other judgment and decision biases — e.g., the endowment effect and non-rational discounting in intertemporal choice — can be accounted for by one’s initial memory query and the resulting constraints on memory search and accessibility (see query theory and the preferences-as-memory framework; Johnson, Haubl, & Keinan, 2007; Weber, Johnson, Milch, Chang, Brodscholl, & Goldstein, 2007).
A common assumption of dual process theories is that controlled cognition (System 2) reflects more rule-based, abstract and decontextualized reasoning whereas more automatic and impulsive cognition (System 1) is driven by associations, personal relevance, and situational-contextual information (cf. fundamental computational bias, Stanovich & West, 2000; but see also Evans, 2008).13 Interestingly, in the current experiment more deliberation was associated with more personalization and contextualization during reasoning — as opposed to abstract rule based expected-value calculations — which was evidenced by more elaborative heuristic search and consideration of more concrete real world implications of choices (for other links between context, abilities, and performance see Delaney & Sahakyan, 2007; Morsanyi & Handley, 2008). Given that elaborative heuristic search accounted for unique variance beyond cognitive abilities, beneficial elaborative search processes may not require an exceptional cognitive capacity or skill. Instead, superior risky decision performance may partially reflect a cognitive style that is typical of (but not necessarily limited to) individuals with higher working memory span. Such a metacognitive style could generally bring more world knowledge to bear on many problems and thus may be less prone to compartmentalization and impulsive choice (Baron, 1985; Stanovich and West, 2000). Additionally or alternatively, these cognitive style factors may be driven by more crystallized knowledge or skill mechanisms. For example, more numerate individuals could derive more affective meaning from the consideration of probabilities and the comparison of options (Peters et al., 2006, Experiment 4), which could motivate more elaborative search.
Broadly, our results are consistent with general notions of reflectiveness suggesting that cognitive abilities are associated with more careful, thorough, and elaborative — but not necessarily normative — cognition (Baron, 1985). Our results further suggest that early selection cognitive control mechanisms may play a role in reflectiveness and superior task performance. Indeed, individuals who score higher on cognitive ability measures are known to spend more time preparing for tasks (Sternberg, 1977) and also more elaborately and strategically encode information, deliberatively building cognitive representations that better support subsequent task performance (Baron, 1978; Cokely et al., 2006; Ericsson & Kintsch, 1995; Hertzog & Robinson, 2005; McNamara & Scott, 2001; Vigneau, Caissie, & Bors, 2005). However, we caution against an interpretation that higher performing individuals (or better decision processes) always search or reflect more (for a discussion of “less is more” in decision making see Gigerenzer et al., 1999). Research unambiguously demonstrates that abilities and expertise are associated with adaptive cognition, such that superior performers will tend to rely on less elaborative search when it is advantageous (Bröder, 2003; Ericsson, Prietula, & Cokely, 2007; Fasolo, Misuraca, & McCelland, 2003, Mata, Schooler, & Rieskamp, 2007; Shanteau, 1992).
Expected value was a reliable as-if choice outcome model. Yet process data indicated that even in highly simplified lotteries expected value was only an as-if model, which showed little relation to actual cognitive processes (Payne & Braunstein, 1978). The priority heuristic also proved to be an inaccurate process (and choice-outcome) model. This limitation may reflect the large individual differences in elaborative search elicited by the current task environment. These data provide further evidence on the limitations and boundary conditions of the priority heuristic (Birnbaum, 2008; Hilbig, 2008; Johnson et al., 2008). It should be noted that this limitation is apparent only because the priority heuristic makes very exact predictions at both the cognitive process and choice-outcome levels, which is a useful and unique feature among risky choice models. Results indicate that more precise process modeling of risky choices with the priority heuristic or another computational model would require at least one parameter that creates variation in search and stopping rules. However, accurate modeling of psychologically plausible mechanisms for the regulation of heuristic search will require greater specification and research at the intersection of task environments and cognitive capacities (Bröder, 2003; Gaissmaier & Schooler, 2008; Gaissmaier, Schooler, & Rieskamp, 2006; Gaissmaier, Schooler, & Mata, 2008; Schooler & Hertwig, 2005).
People higher in working memory span, cognitive reflectiveness, and those with greater skill in comprehending and transforming probabilities often made choices consistent with expected value; however, protocol analyses revealed that they did not commonly use expected-value calculations to arrive at those choices (Payne & Braunstein, 1978; Payne et al., 1993). Instead, cognitive abilities were related to relatively simple yet elaborative heuristic search processes. The results accord with examples showing that good decisions can be made with simple processes (Gigerenzer & Goldstein, 1996; Gigerenzer et al., 1999), although results also provide additional evidence that even heuristic search processes can require conscious, deliberative efforts (Simon, 1990). The current results serve as a reminder that individual differences cannot be ignored by judgment and decision researchers as majority choice does not necessarily reflect a single decision process that can be accurately assessed or modeled at the level of group means (Cokely & Feltz, 2009; Feltz & Cokely, 2008; Regenwetter et al, 2008).
Theoretically, our results indicate that the relationship between cognitive abilities and superior risky choices can reflect differences in relatively simple yet elaborative heuristic-type processes. Nevertheless, the cognitive and metacognitive dynamics that regulate search and stopping are not well understood. The current data provide some indication that these dynamics are likely to be complex and multiply determined, potentially reflecting the influence of early selection cognitive control processes. Further research is needed to identify the variety of mechanisms that give rise to individual differences in decision performance. Critically, a higher fidelity understanding of these mechanisms will require theoretical models to address the interplay of (1) individual differences (e.g., abilities, traits, motivation, expertise), (2) cognitive processes, and (3) the environmental factors that shape strategy selection and efficacy (Alter, Oppenheimer, Epley, & Eyre, 2007; Botella, Pena, Contreras, Shih, & Santacreu, 2009; Galesic, Garcia-Retamero, & Gigerenzer, 2008; Karlsson, Juslin, & Olsson, 2008; Payne et al, 1993; Reiskamp & Otto, 2006; Rieskamp & Hoffrage, 2008; Simon, 1990).
Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: Metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology-General, 136, 569–576.
Bailey, H., Dunlosky, J., & Kane, M. J. (2008) Why does working memory span predict complex cognition? Testing the strategy affordance hypothesis. Memory & Cognition, 36, 1383–1390.
Baron, J. (1978). Intelligence and general strategies. In G. Underwood (Eds.). Strategies of information processing, (pp. 403–450). Academic Press.
Baron, J. (1985) Rationality and intelligence. New York, NY: Cambridge University Press.
Baron, J. (1990). Reflectiveness and rational thinking: Response to Duemler and Mayer (1988). Journal of Educational Psychology, 82, 391–392.
Botella, J., Pena, D., Contreras, M. J., Shih, P. C., & Santacreu, J. (2009). Performance as a function of ability, resources invested, and strategy used. Journal of General Psychology, 136, 41–69.
Brandstätter, E., Gigerenzer, G., & Hertwig, R. (2006). The priority heuristic: Making choices without trade-offs. Psychological Review, 133, 409–432.
Brandstätter, E., Gigerenzer, G., & Hertwig, R. (2008). Risky choice with heuristics: Reply to Birnbaum (2008), Johnson, Schulte-Mecklenbeck, and Willemsen (2008), and Rieger and Wang (2008). Psychological Review, 115, 281–289.
Bröder, A. (2003). Decision making with the “adaptive toolbox”: Influence of environmental structure, intelligence, and working memory load. Journal of Experimental Psychology-Learning Memory and Cognition, 29, 611–625.
Birnbaum, M. H. (2008). New tests of cumulative prospect theory and the priority heuristic: Probability-outcome tradeoff with branch splitting. Judgment and Decision Making, 3, 304–316.
Cokely, E. T., & Feltz, A. (2009). Individual differences, judgment biases, and theory-of-mind: Deconstructing the intentional action side effect asymmetry. Journal of Research in Personality, 43, 18–24.
Cokely, E. T., Kelley, C. M., & Gilchrist, A. L. (2006). Sources of individual differences in working memory: Contributions of strategy to capacity. Psychonomic Bulletin & Review, 13, 991–997.
De Neys, W. (2006). Dual processing in reasoning - Two systems but one reasoner. Psychological Science, 17, 428–433.
De Neys, W., & Glumicic, T. (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106, 1248–1299
Dougherty, M. R. P., & Hunter, J. E. (2003). Hypothesis generation, probability judgment, and individual differences in working memory capacity. Acta Psychologica, 113, 263–282.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working-memory. Psychological Review, 102, 211–245.
Ericsson, K. A., Prietula, M. J., & Cokely, E. T. (2007). The making of an expert. Harvard Business Review, 85, 114–121.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251.
Evans, J. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Reviewof Psychology, 59, 255–278.
Fasolo, B., Misuraca, R., & McClelland, G. H. (2003). Individual differences in adaptive choice strategies. Research in Economics, 57, 219–233.
Feltz, A., & Cokely, E. T. (2008). Do judgments about freedom and responsibility depend on who you are? Personality differences in intuitions about compatibilism and incompatibilism. Consciousness and Cognition. doi: 10.1016/j. concog.2008.08.001.
Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 24–42.
Galesic, M., Garcia-Retamero, R., & Gigerenzer, G. (2008). Using Icon Arrays to communicate medical risks: Overcoming low numeracy. Health Psychology. doi: 10.1037/a0014474
Gaissmaier, W., & Schooler, L. J. (2008). The smart potential behind probability matching. Cognition, 109, 416–422.
Gaissmaier, W., Schooler, L. J., & Rieskamp, J. (2006). Simple predictions fueled by capacity limitations: When are they successful? Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 966–982.
Gaissmaier, W., Schooler, L. J., & Mata, R. (2008). An ecological perspective to cognitive limits: Modeling environment-mind interactions with ACT-R. Judgment and Decision Making,3, 278–291.
Gigerenzer, G., & Todd, P. M. and the ABC Research Group (1999). Simple heuristics that make us smart. Oxford: Oxford University Press.
Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103, 650–669.
Guida, A., Tardieu, H., & Nicolas, S. (2008). The personalisation method applied to a working memory task: Evidence of long-term working memory effects. European Journal of Cognitive Psychology, doi: 10.1080/09541440802236369.
Hertzog, C., & Robinson, A. E. (2005). Metacognition and intelligence. In O. Wilhelm & R. W. Engle (Eds.), Understanding and measuring intelligence, pp. 101–123. London: Sage.
Hilbig, B. E. (2008). One-reason decision making in risky choice? A closer look at the priority heuristic. Judgment and Decision Making, 3, 457–462.
Hsee, C. K., & Rottenstreich, Y. (2004). Music, pandas, and muggers: On the affective psychology of value. Journal of Experimental Psychology-General, 133, 23–30.
Jacoby, L. L., Kelley, C. M., & McElree, B. D. (1999). The role of cognitive control: Early selection versus late correction. In S. Chaiken & Y. Trope, (Eds.), Dual-Process Theories in Social Psychology, pp. 383–402. New York, US: The Gilford Press.
Jacoby, L. L., Shimizu, Y., Daniels, K. A., & Rhodes, M. G. (2005). Modes of cognitive control in recognition and source memory: Depth of retrieval. Psychonomic Bulletin & Review, 12, 852–857.
Johnson, E. J., Haubl, G., & Keinan, A. (2007). Aspects of endowment: A query theory of value construction. Journal of Experimental Psychology: Learning Memory and Cognition, 33, 461–474.
Johnson, E. J., Schulte-Mecklenbeck, M., & Willernsen, M. C. (2008). Process models deserve process data: Comment on Brandstätter, Gigerenzer, and Hertwig (2006). Psychological Review, 115, 263–272.
Kahneman, D. (2003). A perspective on judgment and choice - Mapping bounded rationality. American Psychologist, 9, 697–720.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–291.
Kahneman, D., & Frederick, S. (2007). Frames and brains: Elicitation and control of response tendencies. Trends in Cognitive Sciences, 11, 45–46.
Karlsson, L., Juslin, P., & Olsson, H. (2008). Exemplar-based inference in multi-attribute decision making: Contingent, not automatic, strategy shifts? Judgment and Decision Making, 3, 244–260.
Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44.
Mata, R., Schooler, L. J., & Rieskamp, J. (2007). The aging decision maker: Cognitive aging and the adaptive selection of decision strategies. Psychology and Aging, 22, 796–810.
Mann, L., & Ball, C. (1994). The relationship between search strategy and risky choice. Australian Journal of Psychology, 46, 131–136.
McNamara, D. S., & Scott, J. L. (2001). Working memory capacity and strategy use. Memory & Cognition, 29, 10–17.
Morsanyi, K, & Handley, S. J. (2008). How smart do you need to be to get it wrong? The role of cognitive capacity in the development of heuristic-based judgment. Journal of Experimental Child Psychology, 99, 18–36.
Payne, J. W. (1976). Task complexity and contingent processing in decision-making — information search and protocol analysis. Organizational Behavior and Human Performance, 16, 366–387.
Payne, J. W., Bettman, J. R., Coupey, E., & Johnson, E. J. (1992). A constructive process view of decision-making — multiple strategies in judgment and choice. Acta Psychologica, 80, 107–141.
Payne, J. W., Bettman, J. R., Johnson, E. J., 1993. The adaptive decision maker. New York, NY: Cambridge University Press.
Payne, J. W., & Braunstein, M. L. (1978). Risky choice — examination of information acquisition behavior, Memory & Cognition, 6, 554–561.
Payne, J. W., Samper, A., Bettman, J. R., & Luce, M. F. (2008). Boundary Conditions on Unconscious Thought in Complex Decision Making. Psychological Science, 19, 1118–1123.
Peters, E. & Levin, I. P. (2008). Dissecting the risky-choice framing effect: Numeracy as an individual-difference factor in weighting risky and riskless options. Judgment and Decision Making, 3, 435–448.
Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006). Numeracy and decision making. Psychological Science, 17, 407–413.
Regenwetter, M., Grofman, B., Popova, A., Messner, W., Davis-Stober, C. & Cavagnaro, D. (2008). “Behavioral social choice: A status report.” Philosophical Transactions of the Royal Society B: Biological Sciences. doi: 10.1098/rstb.2008.0259
Reisen, N., Hoffrage, U., & Mast, F. W. (2008). Identifying decision strategies in a consumer choice situation. Judgment and decision making, 3, 641–658.
Rettinger, D. A., & Hastie, R. (2001). Content effects on decision making. Organizational behavior and human decision processes, 85, 336–359.
Rettinger, D. A., & Hastie, R. (2003). Comprehension and decision making. In S. L. Schneider & J. Shanteau (Eds.), Emerging perspectives on judgment and decision research, pp. 165–201. New York, NY: Cambridge University Press.
Rieskamp, J., & Hoffrage, U. (2008). Inferences under time pressure: How opportunity costs affect strategy selection. Acta Psychologica, 127, 258–276
Rieskamp, J., & Otto, P. E. (2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135, 207–236.
Schooler, L. J., & Hertwig, R. (2005). How forgetting aids heuristic inference. Psychological Review, 112, 610–628.
Selart, M., Kuvaas, B., Boe, O., & Takemura, K. (2006). The influence of decision heuristics and overconfidence on multiattribute choice: A process-tracing study. European Journal of Cognitive Psychology, 18, 437–453.
Simon, H. A. (1990). Invariants of human behavior. Annual Review of Psychology, 41, 1–19.
Sloman, S. A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 199, 3–22.
Shanteau, J. (1992). How much information does an expert use? Is it relevant? Acta Psychologica, 81, 75–86.
Sternberg, R. J. (1977). Component processes in analogical reasoning.Psychological Review, 84, 353–378.
Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal of Experimental Psychology: General, 127, 161–188.
Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate. Behavioral and Brain Sciences, 23, 645–665.
Stanovich, K. E., & West, R. F. (2008). On the relative independence of thinking biases and cognitive ability. Journal of Personality and Social Psychology, 94, 672–695.
Thomas, R. P., Dougherty, M. R. P., Sprenger, A. M., & Harbison, J. I. (2008). Diagnostic hypothesis generation and human judgment. Psychological Review, 115, 155–185.
Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory & Language, 28, 127–154.
Tversky, A. & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1130.
Vigneau, F., Caissie, A. F., & Bors, D. A. (2005). Eye-movement analysis demonstrates strategic influence on intelligence. Intelligence, 34, 261–272.
Weber, E. U., Johnson, E. J., Milch, K. F., Chang, H., Brodscholl, J. C., & Goldstein, D. G. (2007). Asymmetric discounting in intertemporal choice: A query-theory account. Psychological Science, 18, 516–523.
Experimental choice options which were presented randomly, once as a gain and once as a loss.
Certain Value Risk odds Risky Value 50 50% 400 225 50% 375 300 80% 2000 125 30% 900 500 70% 600 275 20% 900 100 3% 7000 50 5% 4000 150 5% 2000 200 1% 3000 40 50% 320 270 50% 450 240 80% 1600 150 30% 1080 400 70% 480 330 20% 1080 80 3% 5600 60 5% 4800 120 5% 1600 240 1% 3600
This document was translated from LATEX by HEVEA.