Approximating rationality under incomplete information: Adaptive inferences for missing cue values based on cue-discrimination

Judgment and Decision Making, Vol. 9, No. 2, March 2014, pp. 129-147

Approximating rationality under incomplete information: Adaptive inferences for missing cue values based on cue-discrimination

Marc Jekel*   Andreas Glöckner#  %   Arndt Bröder§   Viktoriya Maydych§

In a highly uncertain world, individuals often have to make decisions in situations with incomplete information. We investigated in three experiments how partial cue information is treated in complex probabilistic inference tasks. Specifically, we test a mechanism to infer missing cue values that is based on the discrimination rate of cues (i.e., how often a cue makes distinct predictions for choice options). We show analytically that inferring missing cue values based on discrimination rate maximizes the probability for a correct inference in many decision environments and that it is therefore adaptive to use it. Results from three experiments show that individuals are sensitive to the discrimination rate and use it when it is a valid inference mechanism but rely on other inference mechanisms, such as the cues’ base-rate of positive information, when it is not. We find adaptive inferences for incomplete information in environments in which participants are explicitly provided with information concerning the base-rate and discrimination rate of cues (Exp. 1) as well as in environments in which they learn these properties by experience (Exp. 2). Results also hold in environments of further increased complexity (Exp. 3). In all studies, participants show a high ability to adaptively infer incomplete information and to integrate this inferred information with other available cues to approximate the naïve Bayesian solution.


Keywords: incomplete information, inference mechanisms, discrimination rate, adaptive decision making, rationality.

1  Introduction

Many states of the world are not directly observable and can only be inferred from cues in the environment that probabilistically relate to the object of inference. The psychological mechanism of inferences based on probabilistic cues have been investigated in various domains, including distance perception (e.g., Brunswik, 1944), personality assessment in social interactions (e.g., Funder, 1996; Gosling, Ko, Mannarelli, & Morris, 2002), market prediction in an economic setting (e.g., Borges, Goldstein, Ortmann, & Gigerenzer, 1999), legal judgments (e.g., Holyoak & Simon, 1999; Glöckner & Engel, 2013), diagnostic decision making (e.g., Swets, Dawes, & Monahan, 2000), and many more. Probabilistic inference can be a difficult task, potentially involving multiple complicating factors. Besides the empirically well investigated challenge that individuals face to select and/or (appropriately) weight and integrate information from various cues, one less intensely investigated complicating factor is that cue information can be incomplete in that potentially important cue values are missing or unknown.

Suppose, for example, that the police must determine which of two suspects, Hans or Bob, has more likely stolen money from a company’s safe, in order to decide which suspect should be the focus of investigations. Pieces of evidence (cues) for making this judgment are incomplete. For some cues, information will be available for one of the suspects but missing for the other. A pen might have been lost by the culprit at the site of crime and a roommate of Hans is certain that Hans does not own such a pen. For Bob it is unknown whether he owned such a pen or not. How would the police treat this missing cue information? One possibility would be to ignore the pen altogether. Another possibility would be to assume that Bob also does not own the pen in that missing cue values are always replaced by negative instances in case of doubt. The police could, however, also rely on more complex reasoning processes to infer the missing cue value. The officer might take into account whether it is a rare pen or not, that is, the base-rate that somebody owns this pen. If it is a very rare pen she might infer that Bob also does not own such a pen. Still, if the officer knows that with a high likelihood the pen comes from one of the two suspects, she will most likely infer that the pen belonged to Bob. Hence, missing information would be inferred from the knowledge that cue values are dependent in that a negative cue value for one suspect implies a positive cue value for the other.

On the one hand, assuming the decision to be important, one might consider that the officer does not just ignore the missing information but aims to infer missing information from the properties of the decision environment in a kind of second-order probabilistic inference. On the other hand, however, it could be argued that implementing such an additional second order inference to the incomplete information would dramatically increase the cognitive effort to solve the task. Considering that deliberate reasoning is limited by cognitive constraints (Simon, 1956) and that in complex tasks the available information might already occupy working memory capacity completely, simply ignoring this information seems psychologically plausible as well. In this paper, we investigate people’s adaptive usage of second order probabilistic inference to incomplete information in complex probabilistic inferences.1

Empirical results indicate that people use a variety of mechanism for inferring missing cue values (see Garcia-Retamero & Rieskamp, 2008, 2009, for overviews). Several studies show that people tend to infer missing cue values as negative information when the context suggests that information is not provided on purpose, such as in a job interview or in the marketing of a product (Huber & McCann, 1982; Jaccard & Wood, 1988; Johnson & Levin, 1985; Lim & Kim, 1992; Stone & Stone, 1987; Simmons, & Lynch Jr., 1991; Yates, Jagacinski, & Faber, 1978). Other studies (Burke, 1995; Ford & Smith, 1987; Körner, Gertze, Bettinger, & Albert, 2007), however, show that people infer missing cue values from other available information and, for example, make inferences from a high price to a high quality of a product (Levin, Johnson, & Faraone, 1984). Still other studies show that the framing of the decision situation (e.g., as involving gains or losses) results in more inferences of positive or negative information (Highhouse & Hause, 1995; Levin, Johnson, Russo, & Deldin, 1985). Even other studies show that people replace cue information by average past values (Meyer, 1981; Slovic & MacPhillamy, 1974) or infer incomplete information by similarity of the current decision with decisions in the past (Sanbonmatsu, Kardes, Posavac, & Houghton, 1997).

Taking into account these findings, Garcia-Retamero and Rieskamp (2008, 2009) proposed that people make inferences about incomplete information that are adapted to the respective structure of the environment: when it is more likely for negative information to be missing than for positive information, it is rational to infer a negative cue value (Garcia-Retamero & Rieskamp, 2008) and people do increasingly use this inference mechanism (Garcia-Retamero & Rieskamp, 2009). When such a contingency is absent, people use other inference mechanism instead, such as replacing missing cue values by the average valence of information in the environment (i.e., the base-rate of positive or negative information). Thus, people do not use a single inference mechanism but choose the inference mechanism that fits the decision environment best.

We extend this work by investigating whether people also use the discrimination rate of a cue as an inference mechanism for missing cue values. We derive conditions under which inferences based on the discrimination rate are particularly successful. We thereby provide a map with the characteristics of the decision environment for identifying regions of rationality for inferences from incomplete information (Hogarth & Karelaia, 2006). We then report three studies that ask whether people use discrimination rate when it is a valid inference mechanism and adaptively switch to other inference mechanisms if necessary.

1.1  The discrimination rate of a cue as an inference mechanism for incomplete cue information

The discrimination rate DR of a cue can be formally defined as the relative number of decisions in which a cue makes a distinct prediction for or against an option (Gigerenzer & Goldstein, 1996). For example, a binary cue that speaks for or against one of two options in 90 decision trials (i.e., + and/or +) and speaks for or against both options in the remaining 10 trials (i.e., + +, ) has a discrimination rate of DR = 90/100 = .90. The importance of the discrimination rate of a cue for implementing a frugal information search has been intensely discussed in the literature (Bröder & Newell, 2008; Gigerenzer & Goldstein, 1999; Newell, Rakow, Weston, & Shanks, 2004; Rakow, Newell, Fayers, & Hersby, 2005). Independent of the cue-validity—that is, how often a cue points towards the better option in the trials where it discriminates—the informative value of a cue decreases if it only rarely discriminates between options.2 When the search for cues is costly, people should and actually also do take the discrimination rate of cues into account and search for cues with low discrimination rates less often (Newell, Rakow, Weston, & Shanks, 2004).

In the current study, we examine the use of the discrimination rate in second-order inferences for incomplete cue values. In situations in which cue information is only partially available, in that a positive or negative cue prediction is available for one option but no information is available for the other option, the discrimination rate of a cue can potentially be used to infer an incomplete cue value. More formally, when a cue has a consistent discrimination rate DR > .5 and the available cue value for one option is positive, it is more likely that the missing cue value is negative than positive (and vice versa for a negative available cue value). In contrast, when the discrimination rate DR of a cue is below .5, it is more likely that the missing cue value has the same valence as the available cue value as compared to having a different value. That is, as long as the discrimination rate is different from .5 (i.e., discrimination in half of the decision trials), the discrimination rate can be used to infer incomplete information.

1.2  Success of using the discrimination rate of cues for inferring incomplete cue information

As shown in the previous section, the discrimination rate of a cue can be used to infer incomplete information in environments with a discrimination rate different from .5, that is when the cue values from the same cue are not independent. Expected success rates of using discrimination rates to infer incomplete information are dependent on the structure of the environment as derived in Appendix A. The analysis reveals that the probability for a correct inference of a missing cue value increases with the absolute difference of the discrimination rate of a cue from .5. Hence, the success of inferences based on the discrimination rate increases when discrimination rates approach 0 or 1, meaning that the dependency between cue values increases. To assess the potential value of using discrimination rates as compared to other inference mechanisms such as replacing incomplete information proportional to the base-rates of cue values (BR) or replacing missing cue values always with a negative or positive cue value (+ vs. in Figure 1), we analyzed the success rates of those inference mechanisms in various environments. Since the accuracy of the latter two inference mechanisms is dependent on the relative frequency of positive and negative cue values in trials for which the cue does not discriminate in the environment, we compared the inference mechanisms (Figure 1) using four exemplary levels of this factor (i.e., extremes with pos = .5 and pos = 1 and two medium values with pos = .7 and pos = .8 to illustrate the relation between variables). Whenever positive and negative cue values are equally frequent in an environment, all inference mechanisms discussed in the past literature lead with a probability of .5 to a correct inference which means success at chance level (Figure 1, upper left panel). Stated differently, in such environments all previously discussed inference mechanisms provide no information gain beyond guessing and are thus useless. In contrast, inferences based on discrimination rates can still be very successful and their success is better than the success of all other strategies for DR <> .5 (Figure 1, upper left panel).


Figure 1: Probability of a correct inference pc (y-axis) for each inference mechanism—that is, discrimination rate (DR), base-rate (BR), positive (+), negative ()—as a function of the discrimination rate (x-axis) of binary cues and four different levels (.5, .7, .8, 1) for the relative frequency of positive cue values (pos) in decision trials with no discrimination. The decision environment of the lower right graph is used in all studies: Each of the three experimental conditions (base-rate high, discrimination rate and base-rate equally high, discrimination rate high) are indicated by a vertical dotted black line. Yellow shaded areas indicate environments in which there is at least one inference mechanism performing better than the discrimination rate. Note: In the upper left figure, all lines for inference mechanisms except for the discrimination rate overlap at pc = .5, that is, those inference mechanisms perform at chance level.

When the frequency of positive cue values for indiscriminate cases increase (and thus the frequency of negative cue values decreases), the probability for a correct inference when inferring a positive cue value and when inferring a cue value in proportion to the base-rate of positive information increase (upper right panel and lower panels, Figure 1).3 Note, however, that using discrimination rates leads to most successful inferences of incomplete information compared to the three other mechanisms for a large range of discrimination rates. The shaded yellow regions in Figure 1 indicate the area in which an inference based on the discrimination rate is outperformed by at least one of the alternative mechanisms. Note also, whenever the discrimination rate of a cue in the environment is above .67, the inferences based on discrimination rates strictly outperform all other mechanisms (white regions in Figure 1). This is due to the fact that discrimination rates and base-rates are not independent and that the former restrict the latter (Appendix A). That is, the discrimination rate restricts the range of the base-rate (i.e., the higher the discrimination rate, the less extreme the base rate). Thus, the analysis shows that the use of the discrimination rate is a powerful and potentially successful inference mechanism for missing cue values in many decision environments.

Given that using the discrimination rate is such a powerful inference mechanism for missing cue values in many environments, we hypothesize that people are sensitive to discrimination rates when making inferences about missing cue values in the context of probabilistic inferences (H1). Additionally, the analysis shows that, compared to other inference mechanisms, inferences based on discrimination rates are successful and therefore adaptive for environments with high or low discrimination rate, and that the mechanism is less successful when the discrimination rate is close to .5 (i.e., all yellow shaded areas in Figure 1). We therefore test the hypothesis—in accordance with prior studies showing that people adapt their inferences to the structure of the environment (Garcia-Retamero & Rieskamp, 2009)—that people adaptively use the discrimination rate only when the discrimination rate is high and that people base their inference concerning incomplete information on other properties of the environment such as base-rates if this is not the case (H2).

2  Experiment 1: Probabilistic inferences with incomplete information in 4-cue environments

In the first study, we tested both hypotheses by investigating whether participants rely on discrimination rates as inference mechanisms for incomplete information at all (H1) and whether they adapt their inference mechanism for missing cue values to the properties of the environment (H2). This adaptation should enable individuals to maintain a high performance with respect to the rational solution even in varying environments, so that people who use the inference mechanism that is adaptive to the environment should also show higher performance (H3). The manipulation of discrimination rates and base-rates was implemented by direct instruction. Specifically, participants were explicitly informed about the base-rates and discrimination rates of the cues in the environment.


Figure 2: Screenshot of a trial in the first study (translated from German). Displayed are the validities of the experts (first column), base-rates for positive information (second column), discrimination rates (third column), and the binary cue pattern (support indicated by pluses and rejection by minuses) for two stocks with one cue value missing (indicated by the question mark). The order of the columns for the base-rate and discrimination rate was counterbalanced between participants. Cues were always ordered from the most to the least valid cue. A hypothetical participant chooses stock dte and is then asked to mark her confidence in the chosen stock being the more profitable stock.

2.1  Method

2.1.1  Participants

Sixty participants (31 female; mean age = 23.6 years, sd = 3.4) were recruited from the MPI Decision Lab Subject Pool using the online recruiting tool ORSEE (Greiner, 2004). Participants received a show-up fee of 8 €($10.0) and performance contingent payment for each correct choice of up to 5.25 €($6.0).

2.1.2  Materials and design

The basic paradigm was a hypothetical stock market game adapted from previous research (Bröder, 2003). Participants selected the more profitable of two stocks in a series of independent choice trials (Figure 2). Participants were provided with information from four experts, constituting binary cues, that made recommendation concerning whether the profitability of the respective stock was good or bad. The recommendations of each expert could differ between the two options or could also be positive or negative for both of them. To induce incomplete information, in each trial a recommendation of one expert for one of the options was missing and replaced by a question mark. The four experts differed in cue validity defined, as the number of correct decisions in 100 past decision trials in which the cue discriminated between options (Gigerenzer & Goldstein, 1996), with val = {.90, .80, .70, .65}. Cues were presented in order of their validity starting with the most valid cue. Discrimination rate (DR) and the base-rate of positive cue information (BR) were manipulated within participants. In the first condition, the base-rate for experts was high (BR = .73) and the discrimination rate was close to .5 (DR = .55). In the second condition, in contrast, the discrimination rate was high (DR = .90) and the base-rate was close to .5 (BR = .55). In the control condition, both BR and DR were equally high with BR = DR = .67.4 Participants were explicitly provided with all information concerning cue validities, base-rates, and discrimination rates (see Figure 2).5 Base-rates and discrimination rates were the same for all experts and remained constant over all trials within a condition. Conditions were presented in blocks of counterbalanced order with each block containing 35 trials (described in Table B.1 of Appendix B, and Section 2.1.5) presented in random order chosen for each participant.

2.1.3  Procedure

Participants received instructions about the stock market game: the information display and the concepts of cue validities, base-rates and discrimination rates were explained in detail. Participants were instructed that validities, base-rates, and discrimination rates of cues are based on 100 prior decision trials of the stock-market game (see also online supplement for instructions used). To assure understanding, participants solved a set of control questions. If participants answered questions incorrectly, they were instructed to re-read the instruction and try again. Additionally, each new block was briefly introduced by an instruction, and further test questions were administered to make sure that participants were aware of the change in base-rates of positive information and discrimination rates.

After completing three exercise trials, participants made 35 choices in each of the three rounds in the stock market game by mouse-click and subsequently indicated their subjective confidence of having chosen the more profitable stock on a continuous slider with the endpoints completely uncertain and completely certain which were recorded as values of −100 to +100, respectively. The order of the conditions was counterbalanced between participants and the order of decision trials within conditions was randomized for each participant. Participants did not receive feedback concerning the accuracy of their choices during the experiment but they were informed that they will receive a summary feedback and a performance contingent payment of 5 cent for each correct choice.6 Finally, demographic data was recorded and participants were debriefed and paid.

2.1.4  Models

A model of choices in probabilistic inference tasks with incomplete information needs to account for a) how people make second-order inferences for missing cue values and b) how they integrate the available and inferred cue values to select one of the options. To address the first question, we tested five inference mechanisms for incomplete information, including the four mechanisms discussed in the previous literature (Garcia-Retamero & Rieskamp, 2008, 2009: base-rate of positive information, positive, negative, and ignoring) and the additional new mechanism based on the discrimination rate of a cue. Concerning the second question, we included four standard models of information integration for probabilistic inference tasks: the non-compensatory Take-the-best heuristic (Gigerenzer, Todd, & The ABC Research Group, 1999; Todd, Gigerenzer, & The ABC Research Group, 2012), a compensatory weighted additive model (Bröder, 2000; Glöckner & Betsch, 2008a; Reimer & Hoffrage, 2006, 2012), a parallel constraints satisfaction network model (Glöckner & Betsch, 2008b), and a random choice model. The resulting 5 × 4 = 20 inference-integration combinations—in the following referred to as models—are listed and described in detail in Table 1.


Table 1: Description of the inference mechanisms and types of information integration. The 20 models tested result from the combination of each inference mechanism with each information integration mechanism.
#
Inference mechanism
Description
Schematic representation of inferences
1
Base-rate of
positive information
A missing cue value is replaced by the base rate of positive information when the cue made a positive prediction in more than half of the decisions before (i.e., base-rate of positive information for the cue > .5).+1, ? ? = BR
2
Discrimination rateA missing cue value is replaced by a positive (negative) cue value when the present cue value is negative (positive) and the cue made a distinct prediction in more than half of the decisions before (i.e., discrimination rate > .5).+1, ? ? = –1
–1, ?
? = +1
3
PositiveA positive cue value is inferred from a missing value.+1, ? ? = +1
4
NegativeA negative cue value is inferred from a missing value.+1, ? ? = –1
5
IgnoreA cue with a missing value is entirely ignored.+1, ? ignored
#
Information integration
Description
1
Take-the-best (TTB)Cues are ordered from highest to lowest validity; the option suggested by the first cue in the order that differs in cue values for options is chosen.*
2
Weighted additive
(WADD
corr)
Cue values are weighted by chance-corrected validities and summed up for each option; the option with the higher weighted sum is chosen.
3
Parallel constraint
satisfaction network
model (PCS)
Cues and options are represented as nodes in a neural network; validities and cue patterns are represented as net weights attached to the links of the nodes; cues are integrated in a coherence-maximizing iterative decision process; the option with the highest node activation is chosen.
4
RandomAn option is chosen randomly.
Note: In the column schematic representation of inferences, a cue with a present cue value and a missing cue value and the inference () for the missing cue value is displayed for each inference mechanism; BR = the base rate of positive information in the environment (e.g., .73); +1 = positive cue value, –1 = negative cue value, and ? = missing cue value. *Note that for TTB and inferences in accord with the base-rate, a missing cue value and a positive cue value (i.e., ? +) do not lead to a decision and the next most valid cue is inspected whereas a missing cue value and a negative cue value (? –1) lead to a decision for the option with the missing cue value when the base-rate of positive information is above .5.

2.1.5  Selection of differentiating decision trials and dependent measures

Cue-patterns (i.e., Figure 2) were selected to allow for differentiating among the 20 models according to the Euclidian diagnostic task selection method (Jekel, Fiedler, & Glöckner, 2012). That is, potential cue-patterns were assessed, sorted and selected by their ability for differentiating between each pair of models taking into account the most diagnostic patterns first. In total, only seven cue-patterns were necessary to disentangle all models.7 Hence, within each condition we selected seven cue-patterns each so that for each pairwise comparison between models there was at least one type of cue-pattern for which models differed concerning choice prediction. Cue-patterns were repeated five times in each condition to allow for a reliable strategy classification.

Participants were classified as being best described by one of the 20 models according to the Multiple-Measure Maximum Likelihood strategy classification (Glöckner, 2009, 2010; Jekel, Nicklisch & Glöckner, 2010). The method relies on a simultaneous maximum-likelihood analysis of individual choices, decision times, and confidence judgments described in Appendix C. We selected decision trials so that in each condition one indicator cue-pattern was included for which all models, including inference mechanism based on base-rates, predict choices for one option while all models including discrimination rates predict the alternative option.

Our manipulation of base-rates and discrimination rates should be reflected in both measures (which are of course not completely independent, since choice data for the indicator cue-pattern is used in the strategy classification as well). Participants in the condition with a high discrimination rate and lower base-rate of positive cue values are expected to decide more in line with the discrimination rate in the indicator cue-pattern and are also expected to be more often classified as users of models that involve the usage of discrimination rates as compared to other inference mechanisms (and vice versa for the condition with a high base-rate and lower discrimination rate). The condition with an equally high base-rate for positive information and discrimination rate is run to assess if there is a preference for one of the inference mechanisms.

2.2  Results

In agreement with the hypotheses, participants adapt their inference mechanism to the structure of the environment (i.e., base-rates and discrimination rates of cues as given in the instructions): participants show more inferences in accordance with the base-rate for the indicator cue-pattern in the condition with a high base-rate (54%) than in the condition with a high discrimination rate (31%) and they show an intermediate choice proportion (48%) in the condition in which the base-rate and discrimination rate were equal (Figure 3, left panel). The difference in choices between an environment with a high base-rate versus a high discrimination rate is significant, t(59) = 4.23, p < .001, one-tailed, according to a mixed effects model with random intercepts per participant (Gelman & Hill, 2007). Hence, results from the indicator cue-pattern support our second hypothesis that individuals use inference mechanisms adaptively to the structure of the environment.


Figure 3: Mean adherence rate of single diagnostic cue-patterns in line with the inference mechanism base-rate for positive information in environments with a high base-rate of positive information, an equal base rate and discrimination rate (only Experiment 1), and a high discrimination rate, for all three studies. Note: Violin plots are displayed: Means are black dots (connected with lines), medians are black thick lines, the borders of the box indicate the lower or upper quartile, whiskers indicate the minimum or maximum data point within 1.5 × the interquartile range, white dots indicate outliers, and shapes around the boxplots indicate the density distribution of the data.

In line with the results from the analysis of the indicator cue-pattern, the Multiple-Measure Maximum Likelihood strategy classification method based on all decision trials reveals that participants use inference mechanisms adaptively to the structure of the environment. Collapsed over all four integration algorithms, in the high base-rate condition most participants are classified either as users of inference mechanisms relying on the base-rate (30%) or a negative cue value (26%) whereas in the environment with a high discrimination rate most participants are classified either as users of the discrimination rate (46%) or a negative cue value (30%) (see Figure 4, upper panel and Table D.1 in Appendix D). Usage of inference mechanisms is significantly influenced by condition according to a Stuart-Maxwell test, χ2 (5) = 18.50, p < .01, providing further support for H2. In the environment with equal discrimination rates and base-rates of positive information, participants show a slight preference for inferences in line with the base-rate of positive information (25% base-rate versus 15% discrimination rate). To double-check the result of the strategy classification, we conducted a global fit test against the saturated model according to Moshagen and Hilbig (2011).8 A considerable proportion of strategy classifications (16%) show a significant misfit of the restricted as compared to the unrestricted model when using p < .05 as misfit criterion. 43% of all misfit identifications concern participants classified as users of negative inferences. Additionally, the relative frequency of participants using the base-rate in an environment with a high base-rate decreases from 30% to 22% when excluding misfits. Thus, except for a less pronounced effect in an environment with a high base-rate, conclusions remain the same when misfits from the analysis are excluded.

Performance of participants was measured as the proportion of choices in accordance with the naïve Bayesian solution. Figure 5 shows the average performance of different subgroups of participants for the different conditions. In both the high base-rate and the high discrimination rate condition participants who use the matching inference mechanism show the best performance. In the environment with high base-rates, participants who rely on base-rates show a higher overlap with the naïve Bayesian solution, t(58) = 4.17, p < .001 (one-tailed). In the environment with a high discrimination rate, participants who rely on discrimination rates show a higher overlap, t(58) = 7.05, p < .001 (one-tailed). Hence, there is support for the third hypothesis, that usage of an inference mechanism in line with the environment also pays off in higher performance.

2.3  Discussion

Experiment 1 investigated choice strategies in probabilistic inferences with incomplete cue information. The analysis revealed that discrimination rates of cues provide a highly efficient inference mechanism for incomplete information. In line with our first hypothesis, a substantial proportion of participants used this inference mechanism, which has not been discussed in the literature before. In line with our second hypothesis, choices in the indicator cue-pattern, as well as results from the Multiple-Measure Maximum Likelihood strategy classification, converge in showing that inference mechanisms are used adaptively. Specifically, inference mechanisms based on discrimination rates are more frequently used in the condition with a high discrimination rate and an uninformative base-rate whereas inference mechanisms based on the base-rate are more heavily relied on in the condition with a high base-rate and an uninformative discrimination rate (i.e., close to .5). Additionally, in line with the third hypothesis, performance with respect to the naïve Bayes solution increases if there is a match between the inference mechanism used and the environmental structure.

Although the results so far provide support for our hypotheses, it might be argued that the results could have been partially due to the fact that information on discrimination rates and base-rates was explicitly provided. It is not clear whether results also generalize to more natural situations in which people are not provided with explicit information but must learn properties of the environment by experience. It might also be questioned whether the results concerning application of effortful second order inferences to incomplete information also hold in more complex environments and hence under conditions of increased working memory load. Each of these concerns is plausible. To test them, we conducted two further experiments in which no explicit information concerning base-rates and discrimination rates was provided: participants had to acquire this information from feedback if they wanted to use it. Additionally, the degree of complexity was increased from four to six cues for the third experiment. Since both studies use the same method and reveal similar findings they are presented jointly.

3  Experiments 2 and 3: Learning the base-rate and discrimination rate of cues in 4- and 6-cue environments

3.1  Method

3.1.1  Participants and Design

Sixty-four (41 female; mean age = 24.4 years, sd = 4.6) and fifty (33 female; mean age = 22.7 years, sd = 3.7) participants took part in Experiments 2 and 3, respectively. They were recruited using the same protocol as before. In both studies participants were compensated by show-up fees (Experiment 2: 17.0 €, $21.0; Experiment 3: 7.0 €, $9.0) and additional performance contingent payment of up to 6.0 €($7.0). The environment structure was manipulated between-subjects. In the first condition, the base-rate for positive information was high and the discrimination rate was close to .5 and therefore relatively uninformative (BR = .73, DR = .55). In the second condition, discrimination rate was high and base-rate was relatively close to .5 (BR = .55, DR = .9).

We again used the stock market game in both studies but increased the number of cues from four to six in Experiment 3. In contrast to the previous experiment, base-rates of positive information and discrimination rates were not given to participants but could be learned in two initial learning phases each consisting of 100 trials of the stock market game.9 The validities of the experts were again explicitly provided (Experiment 2: val = {.90, .80, .70, .65}, Experiment 3: val = {.90, .80, .75, .70, .65, .60}).


Figure 4: Relative frequency of participants in Experiment 1 (upper display) and Experiment 2 and 3 (lower display) for each environment (i.e., high-base rate, high discrimination rate, or equal base-rates and discrimination rates in Experiment 1 only) with a classified inference mechanism (i.e., discrimination rate, base rate, positive, negative, ignore) and an indistinct or unclassified (i.e., unidentified) inference mechanism (i.e., random choice, TTB with positive or base rate inference, unclassified). Purple borders mark the inference mechanism that matches the characteristics of the environment.

3.1.2  Procedure

In contrast to Experiment 1, both studies included an initial learning phase consisting of 200 trials. In the first part of this learning phase, participants made choices in 100 trials and received feedback concerning the missing cue value as well as the better option after each trial. The feedback thus exactly reflected the validities, base-rates and discrimination rates of the respective condition. In the second part of the learning phase, participants again made choices in 100 trials but they were subsequently additionally asked to infer the missing cue value before they received feedback. Participants were informed that they would receive an additional bonus of 1 €($1.0) if they inferred at least 75 out of 100 pieces of incomplete information correctly. The test phase involved 100 trials consisting of 7 types of cue-patterns repeated 10 times and 30 filler cue-patterns. Participants again received 5 Cents for each correct choice in line with the rational model. Within the two learning phases and the the final test phase, decision trials were randomized for each participant. After completing the choice trials, participants’ subjective assessments of base-rates and discrimination rates were measured for both training phases and for the test phase and demographic data was recorded.10 Finally, participants were debriefed and paid.

3.1.3  Models and dependent variables

We tested the same set of 20 models (i.e., 5 inference mechanisms × 4 information integration mechanisms) as before and again used indicator cue-patterns and strategy classification for identifying the inference mechanism used. Hence, both experiments again included indicator cue-patterns that perfectly discriminated between inferences based on the base-rates and inferences based on discrimination rates. Experiment 2 involved one indicator cue-pattern whereas Experiment 3 involved two indicator cue-patterns (which were also repeated 10 times each). In addition to Experiment 1, we also recorded the relative number of inferences of positive and negative cue values in the second learning phase to test in how far participants’ inferences reflect discrimination rates or base-rates.

3.1.4  Selection of cue patterns in the learning and test phase

For the main test phase of Experiment 2 we used the same diagnostic cue patterns as before (Table B.1) and for Experiment 3 we created and selected diagnostic cue patterns with six cues according to the same method as before. The cue patterns for the two preceding learning phases were randomly created but under the restriction that they met the properties of the environment (i.e., BR = .73, DR = .55 versus BR = .55, DR = .9 and the respective validities for each cue). In each trial, one of the cue values was randomly selected for the missing cue value. Two different sets of 100 trial were created and the order of presentation of the sets were counterbalanced between participants.

3.2  Results

In both experiments the results from Experiment 1 concerning all three hypotheses were replicated. In the indicator cue-patterns, participants showed a higher proportion of choices that indicate the usage of the inference mechanism base-rate (as compared to discrimination rate) in the condition with a high base-rate and a low discrimination rate in Experiment 2 (Figure 3, middle) and Experiment 3 (Figure 3, right). This pattern reverses for the condition with a high discrimination rate and a low base-rate. The change in choice proportions between conditions turned out significant in Experiment 2 (t(62) = 6.90, p < .001, one-tailed) and Experiment 3 (t(48) = 2.86, p < .01, one-tailed, in a random slope model with both types of tasks nested within participants; Gelman & Hill, 2007) supporting our second hypothesis stating that individuals adapt their inference mechanism to properties of the environment.

Figure 4 (lower panel) and Table D.1 (Appendix D) report strategy classifications: In the high base-rate condition, the largest group of participants (with identified inference mechanisms) are classified to rely on models involving inferences using base-rates in both experiments. In the high discrimination rate condition, the largest group of participants rely on inferences based on the discrimination rate. The shift in decision strategies between conditions is significant in Experiment 2 (χ2(5, N = 64) = 29.54, p < .001) and Experiment 3 (χ2(5, N = 64) = 17.69, p < .01). In line with our first hypothesis, we again observed a substantial reliance on discrimination rate as inference mechanism when it was appropriate to use it. These findings are noteworthy since information on the discrimination rate was not explicitly provided and people spontaneously relied on discrimination rates based on their experience. Interestingly, the usage of discrimination rate is similar between experiments indicating that increased complexity does not reduce people’s tendency to adaptively use the inference mechanism. One interesting further observation is that the previously observed high proportion of participants that simply replaced missing cue values by negative values decreased almost to zero in Experiments 2 and 3. Hence, in contrast to decisions from explicitly provided information, experience might be better able to drive out mechanisms that are used by default but are maladaptive in the current environment. In both experience-based experiments, almost no participants fail the test between the classified and the saturated model (Moshagen & Hilbig, 2011; 3 participants in Exp. 2 and 1 participant in Exp. 3).


Figure 5: Performance (overlap with the naïve Bayesian solution) for participants classified using the base-rate of positive information, the discrimination rate, or any other inference mechanism for an environment with cues high in base-rate, equal base-rate and discrimination rate, or high discrimination rate for Experiment 1 with four cues and given characteristics of the environment.

Similar to Experiment 1, our third hypothesis stating that a match between the inference mechanism and the environment leads to a higher performance than the usage of a different mechanism is (partially) supported. Participants relying on the base-rate as inference mechanism show (Figure 6) higher performance than participants classified to rely on other mechanisms in the high base-rate condition in Experiment 3 (t(22) = 2.20, p < .05, one-tailed) but not in Experiment 2 (t(32) = 0.02, p = .48, one-tailed). For the high discrimination rate condition, participants who are classified as users of inferences in accordance with the discrimination rate show the highest performance in Experiment 2 (t(28) = 3.24, p < .01, one-tailed) and in Experiment 3 (t(24) = 4.24, p < .001, one-tailed).


Figure 6: Performance (overlap with the naïve Bayesian solution) for participants classified using the base-rate of positive information, the discrimination rate, or any other inference mechanism for an environment with cues high in base-rate or discrimination rate for Experiment 2 with four cues and study 3 with six cues and learned characteristics of the environment.

To allow further testing of our hypothesis of adaptive inferences to incomplete information, in the second part of the learning phase participants did not only indicate choices but they were also asked to infer the missing cue values directly. Results show that participants are sensitive to the structure of the environment in inferring missing cue values not only when this missing information is embedded in a probabilistic inference task but also when the goal is to predict the missing information itself. Participants make more inferences in line with the discrimination rate when the discrimination rate is high as compared to low (Exp. 2: t(62) = 12.6, p < .001, Exp. 3: t(48) = 10.2, p < .001; Figure 7; both one-tailed). Similarly, participants make more inferences in line with the base-rate in the high base-rate condition as compared to the low base-rate condition (Exp. 2: t(62) = 2.9, p < .01, Exp. 3: t(48) = 1.7, p < .05; both one-tailed). The ability to infer incomplete information indicates that participants acquire knowledge concerning the structure of the environment with respect to inferences for incomplete information.


Figure 7: Inferences in line with the discrimination rate or base-rate in the condition with a high base-rate for positive information or high discrimination rate for cues in Experiment 2 with 4 cues and Experiment 3 with 6 cues.

3.3  Discussion

Experiments 2 and 3 closely replicate the results from the first experiment and show that the previous findings are not limited to situations in which information concerning discrimination rates and base-rates is explicitly provided. Results also generalize to more complex decision tasks as shown in Experiment 3. Participants again adapt their inference mechanism to the structure of the environment. Finally, participants’ choices approximate the rational naïve Bayesian solution quite closely and the performance is higher for participants using an inference mechanism that matches the structure of the environment.

4  General discussion

Many situations in which people have to make probabilistic inferences involve incomplete information in that cue values are missing. Previous research indicated that people are rather flexible in how they treat incomplete information and that they adapt their inference mechanisms to the structure of the environment (Garcia-Retamero & Rieskamp, 2009). The main focus of these investigations has been on inferences of missing cue values from base-rates, neglect of missing cue values, or replacement of missing cue values by positive or negative cue values. Based on an analytic approach, we showed that it is more successful in many environments to rely on the discrimination rates of cues to infer missing values instead. Specifically, this mechanism proposes that people infer single pieces of missing cue information based on the available cue value and the degree to which a cue generally tends to make discriminating or equal predictions for the two options compared. We expected that this new inference mechanism based on discrimination rates is adaptively used by participants in environments in which the discrimination rate is a powerful predictor for missing cue values. The results from all three experiments support this proposition. In decisions with explicitly provided information concerning base-rates and discrimination rates but also in environments in which this information has to be spontaneously learned from experience, a considerably proportion of participants relies on discrimination rates to infer missing cue values. These inferred missing cue values are used adaptively to the structure of the environment: discrimination rates are increasingly used in environments in which discrimination rates are particularly informative, and base-rates are not, and vice versa for environments in which base-rates are informative and discrimination rates are less so. Finally, we find that participants who rely on the inference mechanism that is adapted to the respective environment structure tend to show higher performance than participants using other mechanisms.

It is noteworthy that increasing the complexity of the task from four to six cues (i.e., from 8 to 12 pieces for information) did not lead to a shift towards relying on cognitively less demanding inference mechanisms such as simply ignoring missing cue values. The combination of, on the one hand, relatively complex second-order inference to incomplete information and, on the other hand, the application of complex information integration mechanisms for the first order inferences does not seem to tax working memory capacity too much. This independence of complexity therefore speaks for the importance of automatic processes, which have been shown to be only partially sensitive to task complexity (e.g., Glöckner & Betsch, 2012). Finally, in a similar vein, it should be noted that in line with previous findings (e.g., Bröder, 2000; Glöckner & Betsch, 2008b) most participants integrate available and inferred information in a compensatory fashion (i.e., using PCS or WADD, Table D.1). Hence, inferences to missing information and information integration are not cognitively demanding to the extent that people switch to simpler information integration mechanisms.

Our theoretical analysis of the performance of all inference mechanisms in various environments show that inferences in line with the discrimination rate maximize the probability of a correct inference in many environments. This is more so when the discrimination rate is high (i.e., greater than .67). It is an empirical question whether real-world environments more likely consist of discrimination rates that are high or low. Even if cues with uninformative or low discrimination rates (i.e., close to or below .5) are frequent in a decision environment, active search and use of cues with a high discrimination rate (Bröder & Newell, 2008; Newell, Rakow, Weston, & Shanks, 2004) lead to self-tailored decision environments with cues high in discrimination rate. Thus, search for valid cues with high discrimination rates does not only lead to cues that are successful in making correct predictions but also to cues that allow for the application of a powerful inference mechanism when information is partially missing.

Results from the current studies could also have methodological implications for process tracing studies using partially concealed information such as information board and Mouselab. In the information board and its computerized version Mouselab (Payne, Bettman, & Johnson, 1988; Schulte-Mecklenbeck, Kühberger, & Ranyard, 2010), participants are asked to point at information cards to acquire information, and information search is used to test process prediction of decision strategies (e.g., Bröder, 2003; Bröder, Glöckner, Betsch, Link, & Ettlin, 2013; Garcia-Retamero & Rieskamp, 2009; Jekel, 2012; Rieskamp & Otto, 2006). It is thus common to make the assumption that un-revealed information does not influence choices. This implicitly assumes that participants ignore those cues. Given our data, this assumption might be violated, at least in environments with two options only and cues having high (or low) discrimination rates given participants have prior knowledge or acquire knowledge about the properties of the environment in the course of a study. This can lead to wrong conclusions concerning strategy use, particularly in environments in which information search is costly. For example, looking up information partially might not unambiguously indicate the usage of noncompensatory decision strategies and could also result from people using a compensatory weighted additive strategy based on partially concealed but inferred cue values. Following the above argument that these inferences to incomplete information can be done with little cognitive effort, this problem might be quite substantial in some research paradigms and should be kept in mind when interpreting process-tracing measures.

A potential limitation of our research is that we used a selected set of cue-patterns that allowed us to discriminate among models. We cannot completely rule out that this way of cue-pattern selection might influence participants’ inferences for missing cue-values and integration with other information (Rieskamp & Hoffrage, 1999). Furthermore, we used an incentive scheme that rewarded correct inferences for missing information in the learning phase in Experiment 2 and 3, which might have increased participants’ general attention to missing information in their subsequent decisions.

Future studies might address how the process that leads to incomplete information influences the usage of the discrimination rate as an inference mechanism. Information might be missing due to random noise. Alternatively, information might be incomplete on purpose for good or bad intentions. Omission of information can lead to a frugal transmission of information (Gigerenzer, Todd, & The ABC Research Group, 1999) when the omitted information is redundant and/or can be easily inferred by the context (i.e., the other cue value): praising the durability of a product in a consumer-context likely implies, without explicitly mentioning, that the alternative product is worse on this dimension. The downside of this implicit rule in the communication of information (Grice, 1975) is that it can also be used to mislead people by omitting information with the purpose of letting people make invalid inferences about the missing cue value. Such a strategy is most likely more difficult to detect than providing plain wrong information, and such a strategy also disguises the responsibility for misinformation (i.e., the error is made by the receiver’s invalid inference). In how far people are sensitive to the good or bad intentions of strategic use of the discrimination rate in the transmission of information constitutes an interesting question for future research.

References

Borges, B., Goldstein, D. G., Ortmann, A., & Gigerenzer, G. (1999). Can ignorance beat the stock market? In G. Gigerenzer, P.M. Todd, & The ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 59–72). New York: Oxford University.

Bröder, A. (2000). “Take the best—Ignore the rest”: Wann entscheiden Menschen begrenzt rational? Lengerich: Pabst.

Bröder, A. (2003). Decision making with the ’adaptive toolbox’: Influence of environmental structure, intelligence, and working memory load. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29, 611–625.

Bröder, A., Glöckner, A., Betsch, T., Link, D., & Ettlin, F. (2013). Do people learn option or strategy routines in multi-attribute decisions? The answer depends on subtle factors. Acta Psychologica, 143, 200–209.

Bröder, A., & Newell, B. R. (2008). Challenging some common beliefs: Empirical work within the adaptive toolbox metaphor. Judgment and Decision Making, 3, 205–214.

Bröder, A., & Schiffer, S. (2003). Bayesian strategy assessment in multi-attribute decision research. Journal of Behavioral Decision Making, 1, 193–213.

Brunswik, E. (1944). Distal focusing of perception: Size-constancy in a representative sample of situations. Psychological Monographs, 1, 1–49.

Burke, S. J. (1995). The dimensional effects of missing information on choice processing. Journal of Behavioral Decision Making, 8, 223–244.

Ford, G. T., & Smith, R. A. (1987), Inferential beliefs in consumer evaluations: an assessment of alternative processing strategies. Journal of Consumer Research, 14, 363–371.

Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670.

Garcia-Retamero, R., & Rieskamp, J. (2008). Adaptive mechanisms for treating missing information: A simulation study. The Psychological Record, 58, 547–568.

Garcia-Retamero, R., & Rieskamp, J. (2009). Do people treat missing information adaptively when making inferences? The Quarterly Journal of Experimental Psychology, 62, 1991–2013.

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge: University Press.

Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and frugal way: models of bounded rationality. Psychological Review, 103, 650–669.

Gigerenzer, G., & Goldstein, D. G. (1999). Betting on one good reason: The take the best heuristic. In G. Gigerenzer, P. M. Todd, & The ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 75–97). Oxford: Oxford University Press.

Gigerenzer, G., Todd, P. M., & The ABC Research Group. (1999). Simple heuristics that make us smart. New York: Oxford University Press.

Glöckner, A. (2009). Investigating intuitive and deliberate processes statistically: The multiple-measure maximum likelihood strategy classification method. Judgment and Decision Making, 4, 186–199.

Glöckner, A. (2010). Multiple measure strategy classification: Outcomes, decision times and confidence ratings. In A. Glöckner & C. L. M. Witteman (Eds.), Foundations for tracing intuition: Challenges and methods (pp. 83–105). London: Psychology Press & Routledge.

Glöckner, A., & Betsch, T. (2008a). Modeling option and strategy choices with connectionist networks: Towards an integrative model of automatic and deliberate decision making. Judgment and Decision Making, 3, 215–228.

Glöckner, A., & Betsch, T. (2008b). Multiple-reason decision making based on automatic processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1055–1075.

Glöckner, A., & Engel, C. (2013). Can we trust intuitive jurors? Standards of proof and the probative value of evidence in coherence-based reasoning, Journal of Empirical Legal Studies, 10, 230–252.

Gosling, S., Ko, S., Mannarelli, T., & Morris, M. (2002). A room with a cue: Personality judgments based on offices and bedrooms. Journal of Personality and Social Psychology, 82, 379–398.

Greiner, B. (2004). An Online Recruitment System for Economic Experiments. In K. Kremer & V. Macho (Eds.), Forschung und wissenschaftliches Rechnen 2003, GWDG Bericht 63 (pp. 79–93). Göttingen : Ges. für Wiss. Datenverarbeitung.

Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics 3: Speech acts (pp. 41–58). New York: Academic Press.

Highhouse, S., & Hause, E. L. (1995). Missing information in selection: An application of the Einhorn-Hogarth ambiguity model. Journal of Applied Psychology, 80, 86–93.

Hogarth, R. M., & Karelaia, N. (2006). Regions of rationality: Maps for bounded agents. Decision Analysis, 3, 124–144.

Holyoak, K. J., & Simon, D. (1999). Bidirectional reasoning in decision making by constraint satisfaction. Journal of Experimental Psychology: General, 128, 331–336.

Hu, X., & Batchelder,W. H. (1994). The statistical analysis of engineering processing tree models with the EM algorithm. Psychometrika, 59, 21–47.

Huber, J., & McCann, J. (1982). The impact of inferential beliefs on product evaluations. Journal of Marketing Research, 19, 324–333.

Jaccard, J., & Wood, G. (1988). The effects of incomplete information on the formation of attitudes toward behavioral alternatives. Journal of Personality and Social Psychology, 54, 580–591.

Jekel, M. (2012). Validierung des Desired Level of Confidence (Doctoral dissertation). Retrieved from http://hss.ulb.uni-bonn.de/2012/2788/2788.pdf.

Jekel, M., Fiedler, S., & Glöckner, A. (2012). Diagnostic task selection for strategy classification in judgment and decision making: Theory, validation, and implementation in R. Judgment and Decision Making, 6, 782–799.

Jekel, M., Nicklisch, A., & Glöckner, A. (2010). Implementation of the Multiple-Measure Maximum Likelihood strategy classification method in R: Addendum to Glöckner (2009) and practical guide for application. Judgment and Decision Making, 5, 54–63.

Johnson, R. D., & Levin, I. P. (1985). More than meets the eye: The effect of missing information on purchase evaluations. Journal of Consumer Research, 12, 169–177.

Körner, C., Gertze, H., Bettinger, C., & Albert, D. (2007). Comparative judgments with missing information: A regression and process tracing analysis. Acta Psychologica, 125, 66–84.

Lee, M. D., & Cummins, T. D. (2004). Evidence accumulation in decision making: Unifying the “take the best” and the “rational” models. Psychonomic Bulletin & Review, 11, 343–352.

Levin, I. P., Johnson, R. D., & Faraone, S. V. (1984). Information integration in price-quality tradeoffs: The effect of missing information. Memory & Cognition, 12, 96–102.

Levin, I. P., Johnson, R. D., Russo, C. P., & Deldin, P. J. (1985). Framing effects in judgment tasks with varying amounts of information. Organizational Behavior and Human Decision Processes, 36, 362–377.

Lim, J. S., & Kim, J. (1992). Impact of consumers’ confidence in judgements about missing information on product evaluations. Journal of Business Research, 25, 215–229.

Meyer, R. J. (1981). A model of multiattribute judgments under attribute uncertainty and information constraint. Journal of Marketing Research, 18, 428–441.

Moshagen, M. (2010). multiTree: A computer program for the analysis of multinomial processing tree models. Behavior Research Methods, 42, 42–54.

Moshagen, M., & Hilbig, B. E. (2011). Methodological notes on model comparisons and strategy classification: A falsificationist proposition. Judgment and Decision Making, 6, 814–820.

Newell, B. R., Rakow, T., Weston, N. J., & Shanks, D. R. (2004). Search strategies in decision making: The success of “success”. Journal of Behavioral Decision Making, 17, 117–137.

Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988). Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 534–552.

Rakow, T., Newell, B. R., Fayers, K., & Hersby, M. (2005). Evaluating three criteria for establishing cue-search hierarchies in inferential judgment. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 1088–1104.

Reimer, T., & Hoffrage, U. (2006). The ecological rationality of simple group heuristics: Effects of group member strategies on decision accuracy. Theory and Decision, 60, 403–438.

Reimer, T., & Hoffrage, U. (2012). Ecological rationality for teams and committees: Heuristics in group decision making. In P. M. Todd, G. Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality: Intelligence in the world (pp. 335–359). New York: Oxford University Press.

Rieskamp, J., & Hoffrage, U. (1999). When do people use simple heuristics, and how can we tell? In G. Gigerenzer, P. M. Todd, & The ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 141–167). New York: Oxford University Press.

Rieskamp, J., & Otto, P. E. (2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135, 207–236.

Sanbonmatsu, D. M., Kardes, F. R., Posavac, S. S., & Houghton, D. C. (1997). Contextual influences on judgment based on limited information. Organizational Behavior and Human Decision Processes, 69, 251–264.

Schulte-Mecklenbeck, M., Kühberger, A., & Ranyard, R. (Eds.). (2010). A handbook of process tracing methods for decision research: A critical review and user’s guide. New York: Psychology Press.

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.

Simon, H. (1956). Rational choice and the structure of the environment. Psychological Review, 63, 129–138.

Simmons, C. J., & Lynch Jr, J. G. (1991). Inference effects without inference making? Effects of missing information on discounting and use of presented information. Journal of Consumer Research, 17, 477–491.

Slovic, P., & MacPhillamy, D. (1974) Dimensional commensurability and cue utilization in comparative judgment. Organizational Behavior and Human Performance, 11, 172–194.

Stone, D. L., & Stone, E. F. (1987). Effects of missing application-blank information on personnel selection decisions: Do privacy protection strategies bias the outcome? Journal of Applied Psychology, 72, 452–456.

Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.

White, C. M., Koehler, D. J. (2004). Memory & Cognition, 23, 1007–1018.

Yates, J. F., Jagacinski, C. M., & Faber, M. D. (1978). Evaluation of partially described multiattribute options. Organizational Behavior and Human Performance, 21, 240–251.



Appendix A: Probability of a correct inference dependent on the structure of the environment

The probability of a correct inference of a missing cue value depends on the structure of the environment (i.e., a set of decision trials as displayed in Figure 2) as defined by the discrimination rate DR (i.e., relative number of cases in which a cue discriminates between options), the relative frequency of positive information pos and negative information neg in the decision trials where the cue does not discriminate between options (i.e., + + and ). Note that pos and neg are functions of the base-rate of positive cue-values (i.e., BR+) and sample size with:

pos = 
BR+ − .5 × DR
1 − DR
    (1)

and

neg = 1 − pos.     (2)

Under the assumption that there is only one cue value missing for a cue in all trials and that the probability of a missing cue value is equally likely for both types of cue values, the probability for a correct inference when inferring a positive cue value pc+ is:

pc+ = .5 × DR + (1−DR) × pos.     (3)

The probability for a correct inference when inferring a negative cue value pc is:

pc  = .5 × DR + (1−DR) × neg.     (4)

The probability for a correct inference when using the discrimination rate pcDR is:

pcDR = 



      DRif  DR > .50
      .5if  DR = .50
      1 − DRif  DR < .50.
   
      (5)

Finally, the probability for a correct inference when using the base-rate pcBR is:

pcBR  = .5 × DR + (1−DR) ×  (pos × BR+ + neg × BR).

    (6) with:

BR+ = pc+     (7)

and:

BR = pc.     (8)

Table B.1: Choices, decision times, and confidence predictions for the seven target trials for the inference mechanism base-rate of positive information and discrimination rate and all integration mechanisms for the environment with a low base-rate of positive information (.55) and a high discrimination rate (.90) in Experiments 1 and 2.
   Types i of decision tasks
   
1
2
3
4
5
6
7
   
A B
A B
A B
A B
A B
A B
A B
Cue 1 (v = .90)
? +
? +
– +
– ?
– ?
+ ?
+ ?
Cue 2 (v = .80)
+ –
+ –
+ –
+ –
+ +
– +
+ +
Cue 3 (v = .70)
+ +
+ +
+ –
+ +
+ –
+ –
– +
Cue 4 (v = .65)
+ +
+ –
? –
+ –
+ –
+ –
+ –
Inference
Integration Choice Predictions
BR
PCS
 
A
A
A
A
B
A
A
 
TTB
 
A
A
B
B
B
B
B
 
WADDcorr
 
A
A
A
A
A
A
A
DR
PCS
 
B
B
A
B
B
A
A
 
TTB
 
B
B
B
B
B
A
A
 
WADDcorr
 
B
A
A
A
B
A
A
DR, BR
RAND
 
A:B
A:B
A:B
A:B
A:B
A:B
A:B
Inference
Integration Time Predictions (contrasts tTi)
BR
PCS
 
0.406
0.195
0.143
0.068
0.594
0.199
0.117
 
TTB
 
0.143
0.143
0.357
0.357
0.357
0.143
0.643
 
WADDcorr
 
0
0
0
0
0
0
0
DR
PCS
 
0.432
0.545
0.114
0.545
0.136
0.227
0.455
 
TTB
 
0
0
0
0
0
0
0
 
WADDcorr
 
0
0
0
0
0
0
0
BR, DR
RAND
 
0
0
0
0
0
0
0
Inference
Integration Confidence Predictions (contrasts tCi)
BR
PCS
 
0.044
0.462
0.345
0.066
0.538
0.046
0.338
 
TTB
 
0.143
0.143
0.357
0.357
0.357
0.143
0.643
 
WADDcorr
 
0.145
0.613
0.164
0.074
0.387
0.074
0.387
DR
PCS
 
0.260
0.423
0.410
0.423
0.211
0.577
0.330
 
TTB
 
0
0
0
0
0
0
0
 
WADDcorr
 
0.214
0.339
0.161
0.339
0.339
0.661
0.411
BR, DR
RAND
0
0
0
0
0
0
0
Note: Positive cue values are indicated by +, negative cue values by , missing cue values by ?. A:B represents guessing between options. BR = base-rate positive information, DR = discrimination rate.









Appendix B: Model predictions

Choice predictions are derived for each model as described in Table 1. Predictions of decision times are based on the number of computational steps necessary to apply TTB.11 WADDcorr or the number of iterations necessary for PCS (Glöckner & Betsch, 2008a) to reach a coherent solution combined with each type of inference mechanism for missing cue values. Predictions of confidence judgments are based on the validity of the first discriminating cue for TTB, the difference in the weighted sums for each option for WADDcorr, and the difference in activations for option nodes for PCS combined with each inference mechanism for missing cue values. Prediction vectors of decision times and confidence judgments for each model are normalized to prediction contrasts by dividing the centered vector by the range of the centered vector. Predictions for all measures for the seven cue-patterns used for all integration mechanisms are displayed exemplary for the inference mechanism base-rate of positive information and discrimination rate in Table B.1. Note for the first type of tasks that an inference mechanism for missing cue values based on base-rates of positive information leads to a choice for option A, and an inference mechanism for missing cue values based on discrimination rates to a choice for option B independent of the integration mechanism (except for Rand which predicts guessing independent of the inference mechanism). The first type of tasks was used as the indicator cue-pattern in the environment with a high discrimination rate in Experiment 1 and 2.

Appendix C: Primer of the Multiple-Measure Maximum Likelihood strategy classification

To determine which combination of inference for missing cue values and information integration mechanism describes the observed choices and decision times for each participant best, we applied the Multiple-Measure Maximum Likelihood strategy (MM-ML) classification method (Bröder & Schiffer, 2003; Glöckner, 2009, 2010; Jekel, Nicklisch, & Glöckner, 2010). To apply MM-ML, model predictions are derived for each dependent measure (i.e., choices, decision times, and confidence judgments for all three studies; see Appendix B) and the distribution of the data generating process is defined for each measure. That is, choices between two options are assumed to be independent of each other and to stem from a binomial distribution whereas errors for decision times and confidence judgments are assumed to stem from a normal distribution. In MM-ML, model comparisons are based on the comparison between the predictions for each model on choices, decision times, and confidence judgments and the observed behavior.

In more detail, the number of choices njk of type j between two options congruent with model k are assumed to stem from a binomial distribution with a probability of 1 − єk for congruent choices. Observed decision times are winsorized over all participants (2.5 × standard deviation; i.e., approx. 1.2% of the data) to avoid problems with extreme outliers in maximum-likelihood estimation. Additionally, winsorized decision times were regressed on the order of trials presented in a hierarchical mixed-effects linear model with a random intercept and a random slope for order of trials presented for each participant to account for an individual decrease in decision times due to training (Glöckner, 2009, 2010). The resulting residual decision time xTi for trial i is assumed to stem from a normal distribution with a standard deviation σT and a mean µT that is shifted by the prediction scalar tTi and a scaling factor RT in the following way: xTiNT + tTi × RT , σT). Finally, an observed confidence judgment xCi is also assumed to stem from a normal distribution with a standard deviation σC and a mean µC that is shifted by the prediction scalar tCi and a scaling factor RC in the following way: xCiNC + tCi × RC , σC). The maximum-likelihood Ltotal for each participant and each model can then be estimated by maximizing:

Ltotal=
p(njk,
x
 
T,
x
 
C|kkTT,RTCC,RC)=
 
J
j=1


nj
njk


(1−єk)njkєk(njnjk) ×
I
i=1
1
2πσT2
e
(xTi−(µT+tTiRT))2
T2
 
×
I
i=1
1
2πσC2
e
(xCi−(µC+tCiRC))2
C2
 
.

    (9)

Model comparisons are based on the Bayesian information criterion (BIC, Schwarz, 1978) for Nobs trials and Np fitted model parameters that can be calculated by:

BIC=−2ln(Ltotal)+ln(Nobs)Np.     (10)

Participants who show more than єk > 30% strategy inconsistent choices for the most likely model (except for random models with є = 50%) are not classified (see Glöckner, 2009).


Appendix D: Classifications of participants’ inference mechanisms and information integration mechanisms

6inTable D.1: Cross-table with classifications of participants’ inference mechanisms and information integration mechanisms for Experiment 1 to 3 for each model (PCS, WADDcorr, TTB, and RAND) and each experimental condition (BR = base-rate of positive information high, EQ = base-rate of positive information equals discrimination rate, and DR = discrimination rate high) according to the Multiple-Measure Maximum Likelihood strategy classification method based on choices, decision times, and confidence judgments.

Inference mechanismsInformation integration mechanisms
 
PCS
 
WADDcorr
 
TTB
 
RAND
Experiment 1
BR EQ DR
 
BR EQ DR
 
BR EQ DR
 
BR EQ DR
Discr. rate
00 01 08
 
02 04 10
 
05 04 08
 
[gray].7501 06 01
Base rate
01 01 00
 
17 13 11
 
[gray].7501 01 01
 
[gray].75
Positive
00 00 00
 
02 01 01
 
[gray].75
 
[gray].75
Negative
00 04 01
 
11 07 07
 
05 04 10
 
[gray].75
Ignore
00 00 00
 
01 02 00
 
02 00 00
 
[gray].75
01 06 09
 
33 27 29
 
13 09 19
  
 BR EQ DR
Unclassified12 12 02
 Mean SD
Error є.16 .11
Total #60 (within-participants design)
Experiment 2
BR DR
 
BR DR
 
BR DR
 
BR DR
Discr. rate
00 02
 
02 05
 
00 08
 
[gray].7500 00
Base rate
05 00
 
17 06
 
[gray].7500 00
 
[gray].75
Positive
00 00
 
04 00
 
[gray].75
 
[gray].75
Negative
00 00
 
00 01
 
00 01
 
[gray].75
Ignore
00 00
 
02 00
 
01 00
 
[gray].75
05 02
 
25 12
 
01 09
  
 BR DR
Unclassified03 07
 Mean SD
Error є.15 .07
Total #64 (between-participants design)
Experiment 3
BR DR
 
BR DR
 
BR DR
 
BR DR
Discr. rate
00 05
 
02 09
 
01 02
 
[gray].7500 00
Base rate
07 00
 
01 03
 
[gray].7500 02
 
[gray].75
Positive
01 00
 
01 01
 
[gray].75
 
[gray].75
Negative
00 01
 
00 00
 
00 00
 
[gray].75
Ignore
00 00
 
00 00
 
00 00
 
[gray].75
08 06
 
04 13
 
01 04
  
 BR DR
Unclassified11 03
 Mean SD
Error є.18 .06
Total #50 (between-participants design)
Note: Grey areas indicate models that lead to the same prediction (i.e., participants’ inference mechanism cannot be identified for these models). Unclassified participants have a strategy-application-error of є > .30.


*
Georg-August-Universität Göttingen, Georg-Elias-Müller-Institut für Psychologie, Abt. 9: Psychologische Diagnostik, Urteilen und Entscheiden, Goßlerstraße 14, D-37073 Göttingen, Germany, Email: mjekel@psych.uni-goettingen.de
#
Georg-August-Universität Göttingen
%
MPI for Collective Goods, Bonn.
§
University of Mannheim.

This work was supported by Grant GL 632/3-1 and BR 2130/8-1 to Andreas Glöckner and Arndt Bröder from the Deutsche Forschungsgemeinschaft (DFG) as part of the priority program “New Frameworks of Rationality” (SPP 1516).

Copyright: © 2014. The authors license this article under the terms of the Creative Commons Attribution 3.0 License.

1
Here we do not consider the possibility of dealing with incomplete information by putting more effort into information search.
2
For example, the national capital cue in the city size task (Gigerenzer & Goldstein, 1996) always points towards the better option (i.e., validity is 1) but discrimiantes only in 2% of all decision trials (i.e., DR is .02).
3
Note that we plot only pos ≥ .5. For pos ≤ .5 the lines for the probability of a correct inference as positive and negative cue value flip sides.
4
How far these base-rates and discrimination rates are representative of real-world environments (Brunswik, 1944) is an open question: we used extreme values to maximize the potential effect between conditions.
5
For example, the second expert cdf in Figure 2 has a validity of .80 (i.e., % correct = 80), a base-rate of positive cue-values of .55 (i.e., % positive = 55), and a discrimination rate of .90 (i.e., % distinct = 90) based on prior 100 trials. This means that the expert made distinct predictions (+ for one option and − for the other option) in 90 trials and indistinct predictions in 10 trials. Since the base-rate is .55, all indistinct predictions for the 10 trials consisted of positive recommendations for both options (i.e., + +), such that BR+ = (90 + 20)/200 = .55. In the 90 discriminating trials, the expert made 100 × .90 × .80 = 72 correct predictions for the more profitable stock and 90−72 = 18 incorrect predictions for the less profitable stock. See the online supplementary for the instructions and test questions we used to introduce participants to these concepts.
6
A choice was considered to be correct if the option with the higher posterior probability based on the rational computation including all available cues and the optimal inference towards the missing cue value in accordance with naïve Bayes (Lee & Cummins, 2004) was selected. Naïve Bayes computes the odds between stocks given the cue-pattern, under the assumption of independence between cues and equal prior probabilities of having a higher criterion value for both options.
7
This includes all models except for the different combinations of random models with inference mechanisms and TTB with base-rate or positive information as inference mechanism (see Table 1).
8
Specifically, the test investigated whether the model under consideration explains the data significantly worse than a model assuming optimal error levels for each category of choice tasks by comparing log-likelihoods using a G2 statistic (see also Moshagen, 2010, Equations 6–8; and Hu & Batchelder, 1994, Equation 29).
9
Due to a programming error in both experiments in the condition with a high base-rate, discrimination rates in the learning environment slightly varied for cues in a range of .53 to .55 (Exp. 2) and .51 to .57 (Exp. 3) around the targeted discrimination rate (i.e., DR = .55) and base-rates varied between .69 to .72 (Exp. 2) and .70 to .72 (Exp. 3) around the targeted base-rate (i.e., BR = .73); a reanalysis of the data showed that this error slightly affected time and confidence predictions for two of twenty models only, which led to a negligible difference in the maximum-likelihood model-classification for two participants in each experiment.
10
We do not report subjective assessments of base-rates (BR) and discrimination rates (DR) in detail in the article. In general, results show that participants are more sensitive to DR than BR (i.e., subjective assessments of DR are closer to the objective DR in the environment).
11
For example, in case the most valid cue discriminates between options there are four computational steps necessary (read cue value for option 1, read cue value for option 2, compare values, decide); see Glöckner (2009, 2010) for a detailed description on how to derive predictions for all models on all measures used in the experiments.

This document was translated from LATEX by HEVEA.