On the belief that beliefs should change according to evidence: Implications for conspiratorial, moral, paranormal, political, religious, and science beliefs

Does one’s stance toward evidence evaluation and belief revision have relevance for actual beliefs? We investigate the role of endorsing an actively open-minded thinking style about evidence (AOT-E) on a wide range of beliefs, values, and opinions. Participants indicated the extent to which they think beliefs (Study 1) or opinions (Studies 2 and 3) ought to change according to evidence on an 8-item scale. Across three studies with 1,692 participants from two different sources (Mechanical Turk and Lucid for Academics), we find that our short AOT-E scale correlates negatively with beliefs about topics ranging from extrasensory perception, to respect for tradition, to abortion, to God; and positively with topics ranging from anthropogenic global warming to support for free speech on college campuses. More broadly, the belief that beliefs should change according to evidence was robustly associated with political liberalism, the rejection of traditional moral values, the acceptance of science, and skepticism about religious, paranormal, and conspiratorial claims. However, we also find that AOT-E is more strongly predictive for political liberals (Democrats) than conservatives (Republicans). We conclude that socio-cognitive theories of belief (both specific and general) should take into account people’s beliefs about when and how beliefs should change – that is, meta-beliefs – but that further work is required to understand how meta-beliefs about evidence interact with political ideology.

1 Introduction

Rational action requires evidence. Given that beliefs inform action, beliefs ought to be informed by evidence. A longstanding broad perspective on human cognition holds that reason is, at least to some extent, responsible for accurate belief formation (Baron, 2008; Kohlberg, 1969; Piaget, 1932; Stanovich, 2005). However, the human capacity to revise beliefs in the face of conflicting evidence is, charitably, imperfect. Humans are prone to motivated reasoning (Kunda, 1990), identity protective cognition (Kahan et al., 2012), confirmation bias (Nickerson, 1998), myside bias (Perkins, 2019; Stanovich, West & Toplak, 2013), naïve realism (Ross & Ward, 1996), and bias blind spots (Pronin, Lin & Ross, 2002). There is widespread disagreement about the role and consequences of the human capacity to reason.

Various analogies have been used to simplify the various broad perspectives on human thought and, although they may be oversimplifications, they illustrate the disagreement. For example, it has been argued that human reasoning is better characterized by analogy to that of lawyers than philosophers (Haidt, 2012; Haidt, 2001) – that is, the function of human reason is to form arguments to convince others, as is the goal of lawyers, and not necessarily to form accurate beliefs, as is the goal of philosophers (Mercier, 2016; Mercier & Sperber, 2011). Of course, the analogy does not imply that people only reason like lawyers or like philosophers, but rather that the typical characteristics of human cognition are more similar to one frame of thinking than the other. To simplify, some researchers have disputed the common idea that reasoning facilitates sound judgment by pointing to cases (e.g., motivated reasoning) where explicit reasoning actually hurts judgment (Kahan et al., 2012; Kahan, Peters, Dawson & Slovic, 2017).

Relatedly, given evidence that we rely heavily on a number of heuristics and biases (Kahneman, Slovic & Tversky, 1982) and that unconscious processes have an (apparently) widespread impact on our decisions (Bargh & Chartrand, 1999), a prominent perspective is that explicit reasoning and deliberation is just not very effective in the context of powerful intuitions (e.g., Bargh, 1999; Bargh, Schwader, Hailey, Dyer & Boothby, 2012; Dijksterhuis & Strick, 2016; Gigerenzer, 2007; Haidt, 2001). One famous analogy is that human cognition is like an emotional (or intuitive) dog with a rational tail (Haidt, 2001) (or, in a more recent analogy, an intuitive elephant and an analytic rider; Haidt, 2012): That is, our capacity to reason does not effectively override our intuitions and emotional impulses.

These three perspectives can be summarized simplistically in terms of three general claims about the nature of human reasoning: 1) that reasoning prototypically helps make good decisions and come to informed beliefs (“reasoning is helpful”); 2) that reasoning is prototypically ineffective, since intuition dominates human cognition (“reasoning is helpless”); and 3) that reasoning prototypically undermines sound judgment and exacerbates motivated reasoning and (for example) political polarization (“reasoning is hurtful”). Although any of the three accounts may be the best explanation for the underlying psychology behind any given belief/opinion/value, the critical question here is which accounts offers the best broad description of high-level human cognition (i.e., which has the greatest explanatory power across various beliefs/opinions/values).

Although recent work has attempted to mediate between these three broad accounts by investigating individual differences in analytic thinking (e.g., Pennycook, Fugelsang & Koehler, 2015a; Pennycook & Rand, 2019b; Pennycook, 2018), this work is vague on the specific aspects of analytic thinking that support good thinking. Here we contend that people differ in terms of their explicit stance toward whether beliefs ought to change according to evidence and that this has major consequences for what beliefs, opinions, and values that they hold. That is, some may place stronger value in changing their beliefs and taking relevant evidence into account – and thereby (for example) take a stance toward reasoning that is more akin to a philosopher – whereas others may place stronger value in maintaining constancy and the defense of prior beliefs – and thereby take a stance toward reasoning that is more akin to a lawyer. Moreover, this meta-belief may impact what sort of beliefs individuals endorse as adults, indicating that reasoning really does have an impact on intuitive beliefs. The goal of the present work is to investigate these possibilities.

1.1 Is reasoning helpful or helpless?

Dual-process theories of reasoning, which distinguish from autonomous (intuitive) processes and those that are accomplished via some form of deliberative control (De Neys, 2017; Evans & Stanovich, 2013; Pennycook, Fugelsang & Koehler, 2015b; Thompson, Prowse Turner & Pennycook, 2011), typically emphasize how controlled reasoning processes can override (sometimes) incorrect intuitive responses. Although this emphasis does not imply that reasoning and accuracy are synonymous, it does suggest that there are meaningful and important cases where such an association is present (Evans, 2012). To take a recent example, individuals who are more disposed toward thinking analytically (as indexed by the Cognitive Reflection Test; Frederick, 2005) are less likely to fall for fake news regardless of whether it is consistent or inconsistent with their political ideology (Pennycook & Rand, 2019b). There is also evidence that analytic thinking is associated with disbelief in a variety of epistemically suspect beliefs (Pennycook et al., 2015a), such as in paranormal and religious beliefs (Pennycook, Cheyne, Seli, Koehler & Fugelsang, 2012; Pennycook, Ross, Koehler & Fugelsang, 2016; Shenhav, Rand & Greene, 2012), conspiratorial ideation (Swami, Voracek, Stieger, Tran & Furnham, 2014), anti-science beliefs (and specifically rejection of evolution) (Gervais, 2015), and pseudo-profound bullshit receptivity (Pennycook, Cheyne, Barr, Koehler & Fugelsang, 2015). In addition, there is evidence that reliance on intuition is associated with traditional moral values (Pennycook, Cheyne, Barr, Koehler & Fugelsang, 2014; Royzman, Landy & Goodwin, 2014) and conservative political ideology (Jost, 2017) (but perhaps more-so with political apathy; see Pennycook & Rand, 2019a).

In contrast, a large and diverse body of evidence supports the idea that reason is perhaps overrated. For instance, intuitive heuristics are often extremely useful and, in some contexts, may actually be more accurate than reasoned reflection (Gigerenzer, 2007; Gigerenzer, Todd & ABC Research Group, 1999). Illustrative (albeit extreme) examples of this come from research on expertise (Klein, 2008), which shows that naturalistic decision making (e.g., among chess masters or firefighters) allows for very rapid yet extremely accurate choice (Kahneman & Klein, 2009). Furthermore, social psychology in the 1990’s provided numerous examples of the surprising power of intuition (Bargh & Chartrand, 1999; Dijksterhuis & Strick, 2016; Haidt, 2012). Although there have been questions about the replicability of some of these effects (e.g., for so-called “social priming” [Cesario, 2014]), the strong influence of intuition on decision making is not a matter of dispute (Evans, 2008; Kahneman & Klein, 2009).

1.2 Does reasoning undermine sound judgment?

In contrast to the work just reviewed, there is also considerable evidence for motivated reasoning effects (Kunda, 1990) – that is, cases where reasoning actively hurts sound judgment and causes people to become further entrenched in what they already believe (Kahan, 2013). For example, people tend to dismiss information that is inconsistent with their political ideology (Lodge & Taber, 2005; Redlawsk, 2002; Redlawsk, Civettini & Emmerson, 2010; Strickland, Taber & Lodge, 2011) and engage in biased search for information that is supportive of their beliefs (i.e., confirmation bias; Knobloch-Westerwick, Mothes & Polavin, 2017; Nickerson, 1998). In fact, there is evidence that political polarization about contentious scientific issues (such as global warming) is actually greater among individuals who are more intelligent (Kahan et al., 2012; Kahan, Peters, Dawson & Slovic, 2017; Sarathchandra, Navin, Largent & McCright, 2018) and who report having a more actively open-minded thinking style (Kahan & Corbin, 2016; but see Baron, 2017).

A parsimonious broad account of these findings is that individuals engage analytic reasoning processes, not in the service of accuracy, but as a means to protect their identity (Kahan, 2013) or to form convincing arguments (Mercier, 2016). This perspective flips the common conception of human reasoning on its head and suggests that reasoning often makes people more unreasonable. Consistent with this account, a recent meta-analysis indicated that partisan bias effects (motivated reasoning) were equivalent across the political spectrum (Ditto et al., 2019; but see Baron & Jost, 2019). This research indicates that reasoning is typically (or, at least, frequently) used in service of justifying prior beliefs, as opposed to updating them based on the evidence presented. To investigate this issue, we will focus on the idea that individuals who are more prone to engage in reasoning are more (not less) politically polarized. Consistent evidence for increased polarization among highly reflective people would indeed indicate that motivated reasoning is to be expected; to return to an earlier analogy, that humans reason more like lawyers than philosophers.

1.3 Actively open-minded thinking

Despite research showing evidence for motivated reasoning and the power of intuitions, the previously reviewed associations between analytic thinking and various beliefs/values suggests that reasoning is nonetheless used to modify beliefs in everyday life (although other factors are of course involved in determining what people believe). That is, people who are more reflective when they are given a trick question from the Cognitive Reflection Test (CRT) have different beliefs than intuitive people. A parsimonious explanation of this is that the same people who reflect on the CRT also tend to reflect about their beliefs (i.e., they use reason to modify beliefs). Nonetheless, the disposition to engage analytic thinking is not the same as having an actively open-minded stance in general (Baron, 1985; Stanovich & West, 1997) or toward evidence in particular (Baron, 2019; Baron, Scott, Fincher & Metz, 2015). Indeed, analytic thinking may be used to both override intuitions (i.e., to modify or undermine prior beliefs) or to rationalize or bolster intuitions (i.e., to reinforce prior beliefs) (Pennycook, Fugelsang, et al., 2015b).

In the present work, we will focus instead on people’s beliefs about whether beliefs and opinions should change according to evidence. Moreover, we will investigate a wide range of beliefs, values, and opinion together as a way to systematically assess the potential long-term impact of people’s thinking style on what they think.

The idea that some people may not be disposed to using evidence to inform their beliefs has been broached previously. For example, people may differ in their “criteria” for belief; although some hold that evidence and scientific consensus are most important, others believe that “knowledge of the heart” should also be a central consideration (Metz, Weisberg & Weisberg, 2018). Indeed, the actively open-minded thinking scale (AOT) was created to assess (in part) the belief that it is good to seek evidence that may conflict with intuitions (Baron, 2008; Baron et al., 2015; Baron, 1985; Stanovich & West, 1997; see also Price, Ottati, Wilson & Kim, 2015, for a measure based more on self-report) – a tendency that is associated with improved decision making over and above intelligence or cognitive ability (Stanovich & West, 2000; Stanovich & West, 1998). Moreover, much like individual differences in cognitive reflection, high AOT has been linked to skepticism about supernatural claims (Baron et al., 2015; Pennycook, Cheyne, Barr, Koehler & Fugelsang, 2014; Svedholm & Lindeman, 2013) and superstition (Sá, West, & Stanovich, 1999), indicating that the AOT scale may index some aspects of openness to evidence in belief formation and revision. Indeed, Svedholm-Häkkinen and Lindeman (2018) found a “fact resistance” factor within the broader AOT measure that consists of items that ask about beliefs about changing beliefs according to evidence. As noted by Baron (2019), it is this “flexible thinking” dimension that is most central to the concept of AOT. Shortened versions of the AOT scale have also typically focused largely on the belief revision questions (Baron et al., 2015; Haran, Ritov & Mellers, 2013), which further suggests that these items are of particular relevance for the AOT’s predictive validity. Nonetheless, the broad consequences of this meta-belief across a variety of domains has not yet been systematically investigated despite having major relevance for several broad theories of human cognition. We will refer to our subscale simply as actively open-minded thinking about evidence (AOT-E). The items for our scale can be found in Table 1. Our AOT-E scale is not the same as has been used in the past, although some of the items are from previous (longer) versions of the AOT. For further information on how we derived the AOT-E scale from the larger full AOT scale, see the two validation studies presented in the supplementary materials.

1.4 Current work

Is reasoning prototypically helpful, helpless, or hurtful? One possibility that has not yet been broached is that the three perspectives are primarily describing different people. That is, people have different beliefs about whether beliefs should change according to evidence (“meta-beliefs”) and this has consequences for the effectiveness of their reasoning and, therefore, what types of beliefs that they hold. The goal of the present investigation is to determine whether AOT-E is correlated with as wide a variety of beliefs, values, and opinions as is feasible in a single study. If AOT-E is consequential, it should be associated with people’s stances on a number of important issues. To this end, we investigated conspiratorial, moral, paranormal, political, religious, and science beliefs.

2 Study 1

2.1 Method

2.1.1 Participants

American participants were recruited from Mechanical Turk on February 18^th, 2016. We set our goal sample at 350 and over sampled 380 participants (assuming some degree of attrition due to random responding). Only 3 participants responded affirmatively when asked if they responded randomly at any point during the survey and 3 participants did not answer affirmatively when asked if they are fluent in English. The resulting sample (N = 375, Mean age = 35.8) consisted of 216 males and 158 females (1 participant did not indicate their gender).

2.1.2 Materials

Measures were converted into POMP scores, i.e.(raw-min)/(max-min), ranging from 0-100 (Cohen, Cohen, Aiken & West, 1999). Data and materials for all studies are available on OSF: https://osf.io/xqzse/.

AOT-E.

We administered the AOT-E scale that is presented in Table 1. Participants responded on a scale from 1) Strongly disagree to 6) Strongly agree. The AOT-E had strong reliability (α = .87). Participants rated themselves as, on average, willing to change their beliefs according to evidence (M = 69.8, SD = 19.1 – scale ranges from 0–100). Only 19.2% of the participants were at or below the scale midpoint (indicating a resistance to evidence).

Conspiracist ideation.

Participants completed a 15-item general conspiracy beliefs scale (Brotherton, French & Pickering, 2013). The scale included items such as “A small, secret group of people is responsible for making all major world decisions, such as going to war” (α = .97). Responses were made on the following 5-point scale: 1) Definitely not true, 2) Probably not true, 3) Not sure/cannot decide, 4) Probably true, 5) Definitely true.

Paranormal belief.

Participants completed a slightly revised Paranormal Belief Scale (Pennycook, Cheyne, Seli, Koehler & Fugelsang, 2012; Tobacyk, 2004) with the religious belief items excluded (α = .95). The scale consisted of 22 items sampled from 6 categories of supernatural belief (example items in parentheses): Psi (“Mind reading is possible”), Witchcraft (“Witches do exist”), Omens of luck (“Black cats can bring bad luck”), Spiritualism (“It is possible to communicate with the dead”), Extraordinary life forms (“The Loch Ness monster of Scotland exists”) and Precognition (“Astrology is a way to accurately predict the future”). Participants indicated their belief by responding on a 7-point scale from 1) Strongly disagree, to 4) Uncertain, to 7) Strongly agree.

God Skepticism.

Skepticism about God was assessed using the following question: “What sort of God, if any, do you believe in?” and presenting the following options of increasing skepticism (Pennycook et al., 2012; Pennycook, Ross, et al., 2016): 1) A personal God [Theism], 2) God as an impersonal force [Pantheism], 3) A God who created everything, but does not intervene in human affairs [Deism], 4) Don’t know whether or not any Gods exist [Negative Agnostic], 5) Don’t know whether or not any Gods exist and no one else does either [Positive Agnostic], 6) I don’t believe in Gods of any sort [Negative Atheist], and 7) I believe that God does not exist [Positive Atheist].

Moral values.

We used Pennycook, Cheyne, Barr, Koehler, and Fugelsang’s (2014) moral values scale, which consisted of 6 care/fairness (“individualising”) and 4 traditional (“binding”) moral values (Graham et al., 2011). Participants were asked to rate how important the values were to their moral thinking on a 7-point scale from 1) Irrelevant to 7) Extremely Important. Care/fairness values included being kind, supporting the autonomy of others, being helpful, being fair, avoiding harm, and supporting the rights of others (α = .85). Traditional values included showing respect for traditions, being patriotic and loyal, showing respect for legitimate authority, and being pure by avoiding carnal pleasures and disgusting things (α = .80).

Political ideology.

Participants were asked to indicate their stance on social and economic issues separately on scales from 1) Very liberal, to 3) Moderate, to 5) Very conservative. Following Pennycook and Rand (2019a), we computed four political categories based on the convergence between social and economic political ideology: 1) Consistent Liberals, who are liberal/very liberal on both social and economic issues, 2) Consistent Conservatives, who are conservative/very conservative on both social and economic issues, 3) Libertarians, who are liberal/very liberal on social issues but conservative/very conservative on economic issues, and 4) Consistent Moderates who are moderate on both social and economic issues. However, because of the liberal political skew of Mechanical Turk, there were only 60 Consistent Conservatives in our sample compared to 153 Consistent Liberals.

Political opinions.

We also surveyed a range of political opinions (see Table 2). Participants were asked to indicate agreement/ disagreement on a 7-point scale from 1) Strongly disagree to 7) Strongly agree. As is evident from Table 2, three of the items did not correlate particularly highly with political ideology (microaggressions, campus free speech [coded so that support of free speech was counted as conservative], and men experiencing sexism). We therefore created a Conservative Opinions scale (α = .81) using all items except for these three (all items were re-scored so that a high score indicated a more conservative opinion). Participants were also asked to indicate their relative trust in the government on a scale from 1) Strongly Distrust to 5) Strongly Trust (this was also uncorrelated with political conservatism, r = -.07).

Free Market Ideology.

Participants completed a 5-item Free Market Ideology measure (Heath & Gifford, 2006). The scale assesses the belief in the powers of the free market (α = .85). It includes items such as: “An economic system based on free markets unrestrained by government interference automatically works best to meet human needs.” Responses were provided on a 7-point scale from 1) Strongly disagree to 7) Strongly agree.

Science beliefs.

We created a science belief scale based on various contemporary scientific issues. In particular, we selected a number of typical science-related beliefs (Table 3): evolution, anthropogenic global warming, big bang theory, old Earth, and stem cell research. We also attempted to use items that have been associated with “liberal” anti-science attitudes (Table 3): resistance to technology (reverse scored), genetically modified organism (GMO) resistance (reverse scored), vaccines as a cause of autism (reverse scored), and belief in modern medicine. However, as is evident from Table 3, political conservatives were more likely to hold the more anti-scientific stance on every single issue – even issues often associated with political liberalism. Nonetheless, consistent with prior research, there was large variability in terms of how strongly conservatism predicted anti-scientific attitudes (Rutjens, Sutton & van der Lee, 2018). Participants responded on a 7-point scale from 1) Strongly disagree to 7) Strongly agree; however, for our primary analysis, all items were scored such that a higher value meant a more pro-science belief. The full scale had good reliability, α = .84. Participants were also asked to indicate their relative trust in scientists on a scale from 1) Strongly Distrust to 5) Strongly Trust.

Demographics.

Participants were given a demographic questionnaire that included the following items: age, gender, and English proficiency. Social and economic political ideology were included in the demographics questionnaire.

2.1.3 Procedure

Participants either completed the AOT-E at the beginning of the survey or at the end (but before demographics). The presentation order did not change the pattern of results and the aggregate results will therefore be reported. Otherwise, the order of the measures was as follows: 1) conspiracist ideation, 2) paranormal belief, 3) moral values, 4) science beliefs, 5) political opinions, 6) free market ideology, 7) theism, and 8) demographics (including political ideology).

2.2 Results and Discussion

As is evident from Table 4, AOT-E was strongly associated with every other primary measure. Individuals who believe that beliefs should change according to evidence (those high in AOT-E) were: a) less likely to believe conspiratorial, paranormal, and religious (and, specifically, theistic) claims, b) less likely to hold traditional moral values but were more likely to adopt care/fairness moral values, c) less conservative in terms both economic and fiscal ideology (including free market ideology) and across a range of specific political opinions, and d) less likely to hold anti-science beliefs. Gignac and Szodorai (2016) meta-analyzed typical effect sizes across social psychology and found that correlations (r) of .10, .20, and .30 can be considered relatively small, medium, and large, respectively. Using this metric, AOT-E was a remarkably strong predictor of most factors. With the exceptions of care/fairness moral values (r = .26) and the single-item fiscal conservatism (r = −.24), every effect size was above what would be considered large based on empirical norms. The correlation with conservative opinions and pro-science beliefs, in particular (r’s greater than .60), were well above the 95^th percentile (r = .45) in terms of effect size norms for individual differences research in psychology (Gignac & Szodorai, 2016). This overall pattern of result undermines the idea that reasoning is ineffective and is consistent with the general claim that reasoning has major impacts on our beliefs and values.

To further understand the scope of AOT-E’s predictive validity, we also investigated the extent to which it predicted specific political opinions (Table 5). With respect to political opinions, individuals who indicate being more actively open-minded about evidence held broadly liberal political views. Indeed, AOT-E was less predictive for the items that were less strongly associated with political ideology: whether men experience sexism on par with women (AOT-E was significantly associated with disagreement, r = −.24); whether microaggressions are a serious problem in educational contexts (AOT-E was slightly but non-significantly associated with disagreement, r = .07)¹; and whether students should be able to block controversial speakers from giving talks at their university (AOT-E was significantly associated with disagreement, r = .25). The items most strongly associated with political conservatism were most strongly negatively associated with AOT-E.

The pattern of results for individual science belief items (Table 6) was very clear (and plainly in support of the “reasoning helps” perspective): AOT-E was associated with more agreement with scientists, regardless of whether the issue pertained to agreement with a clear scientific consensus (such as around anthropogenic global warming or the big bang) or a disagreement with an anti-scientific belief (such as that GMO’s are unhealthy or that vaccines cause autism). AOT-E was also positively associated with general trust in scientists.

3 Study 2

The results of Study 1 indicate that AOT-E is a very strong predictor of a wide range of beliefs and opinions. There are, however, three key issues that the data from Study 1 leave unresolved. The first pertains to the perhaps implausibly large effect sizes that we found in Study 1. A recent paper by Stanovich and Toplak (2019) raised an important point that pertains to the AOT (and that applies to the AOT-E): When asked about “beliefs”, some individuals may assume that the question is really about religious beliefs. Indeed, Stanovich and Toplak found that the extremely high correlation between AOT and religious beliefs can be partially (but not fully) accounted for using “belief revision” items (that is, the same class of items that make up the AOT-E). Of course, this may be partly due to the possibility that AOT-E plays a major role in belief formation (as we have argued). However, it may also be the case that religious believers are particularly opposed to revising their religious beliefs, but less opposed to revising beliefs in general. Consistent with both of these possibilities, Stanovich and Toplak found that items that used slightly different wording (which did not lead to the religious belief presumption) continued to predict religious belief, but not as strongly. In Study 2, we therefore modified the AOT-E to ask about “opinions” instead of “beliefs” (see Table 7). We also changed the wording of an additional item so that there would be an equal number of standard and reverse-coded items. Participants in Study 2 were either administered the original AOT-E or the revised AOT-E. Our goal was to ask whether the results of Study 1 are robust to variations in AOT-E scale wordings even if effect sizes vary somewhat.

A second drawback of Study 1 is that our sample came from Mechanical Turk and is therefore particularly unrepresentative of political conservatives. This is a notable drawback because the association between AOT-E and political opinions may differ depending on whether the individuals are politically liberal or conservative. As such, in our second study, we collected a sample from Lucid for Academics – a source that provides American samples that are nationally representative on age, gender, ethnicity, and geography (based on quota-matching), and that therefore provides a more even and representative split of liberals and conservatives (Coppock & Mcclellan, 2019; Pennycook & Rand, 2019a).

Third, many of the AOT-E correlates reported in Study 1 have, in previous research, been shown to correlate with performance on the Cognitive Reflection Test (CRT; Frederick, 2005; Pennycook, Fugelsang, et al., 2015a) – a measure intended to assess the broad disposition to think analytically and that also correlates with AOT (Toplak, West & Stanovich, 2011). Thus, in Study 2 we included the CRT to assess the relative predictive strength of AOT-E relative to CRT.

3.1 Method

3.1.1 Participants

American participants were recruited from Lucid for Academics on April 19^th, 2019. We recruited 700 participants, who were randomly assigned to one of two conditions. In total, 751 participants began the study but 60 did not finish. We also removed individuals who responded affirmatively when asked if they responded randomly at any point during the survey (77 from the original AOT-E condition and 76 from the revised AOT-E condition). The resulting sample (N = 539, Mean age = 45.4) consisted of 251 males and 278 females, 1 transgender female, 1 transgender male, 3 trans/non-binary, 4 “not listed”, and 1 who preferred not to answer.

3.1.2 Materials

Measures identical to Study 1.

The following measures were administered as in Study 1: Conspiracist ideation, paranormal belief, God skepticism, moral values, political ideology, political opinions, free market ideology, and trust in scientists. Unlike in Study 1, all of the political opinion items were significantly associated with political ideology (Table 8). We therefore used all of the items to form the political opinions scale (α = .72).

AOT-E.

Participants were either administered the original or the revised AOT-E scale, as outlined in Table 7. Reliability is good for both scales (original: α = .72; revised: α = .74), albeit not as strong as in Study 1. Participants reported being more actively open-minded when asked about opinions (revised scale; M = 65.5, SD = 16.5) than beliefs (original scale; M = 56.7, SD = 17.2), t(537) = 6.07, SE = 1.45, p < .001. Whereas 43% of the sample were at or below the scale midpoint when asked about beliefs, only 20.4% were at or below the scale midpoint when asked about opinions. Thus, although only a minority indicated a resistance to evidence in both conditions, this was more common when asked about beliefs than opinions. This is what would be expected if the conflation of beliefs with religious beliefs was causing some individuals to indicate a resistance to evidence. Alternatively (or in addition), it is possible that people are simply more open to changing opinions (which may be issues of taste/preference) than beliefs (which may refer more to people’s position on issues of apparent fact). At any rate, the revised AOT-E removed the apparent bias against religious individuals (Stanovich & Toplak, 2019).

Cognitive Reflection Test (CRT).

We used a re-worded version (Pennycook & Rand, 2019b) of the three-item CRT (Frederick, 2005). The CRT consists of words problems that cue an incorrect intuitive response and that therefore partially index one’s disposition to engage in reflective reasoning (Campitelli & Gerrans, 2014; Pennycook, Cheyne, Koehler & Fugelsang, 2016; Toplak et al., 2011). The Lucid sample had particularly low accuracy on the CRT (M = .16, SD = .28; i.e., 0.5 out of 3 correct, on average – 70% of the sample got 0 out of 3). As a consequence, reliability was relatively low for the CRT (α = .64).

Religious belief.

In addition to the theism measure used in Study 1, we also included a full religious belief scale (via Pennycook et al., 2016). For this, participants were asked to indicate their degree of belief in the following supernatural religious claims: afterlife, heaven, hell, miracles, angels, demons, soul, devil/Satan, and God. Participants responded on a 5-point scale from 1) Strongly disagree to 5) Strongly agree. The religious belief scale had excellent reliability (α = .95). Unfortunately, there was a significant amount of missing data (N = 90) for the religious belief scale – perhaps because it was the only scale that was administered using a matrix responding format (this was done because our intention was to administer the scales identically as they have been administered in past research).

Science beliefs.

We attempted to expand our science belief questionnaire by adding additional items for which political liberals might be expected to have more anti-scientific stances. Specifically, we asked about the following (in addition to the items from Study 1; see Table 9): the heritability of human intelligence, the role of genetics in success, “detox” therapies, and nuclear power. However, as is evident from Table 9, the only anti-scientific stance that was more common among political liberals was opposition to nuclear power. Nonetheless, unlike Study 1, many of the issues (6 out of 13) did not significantly correlate with political ideology. At any rate, the full scale had acceptable reliability (α = .72).

Political party.

In additional to the political ideology questions that were administered in Study 1, we also asked participants to indicate which political party they most strongly affiliate with: Democrat, Republican, Independent, Other. The sample was fairly politically balanced: 37% Democrat, 31% Republican, 29% Independent, and 3% “other”. We also asked them who they voted for in the 2016 Presidential Election, about favorability toward Donald Trump, and to indicate the likelihood that they would vote for Trump in the 2020 Presidential Election. These measures, along with social and economic political ideology, were included in the demographics section of the survey.

Demographics.

Participants were given a demographic questionnaire that included the following items: age, gender, English proficiency, education, income, and ethnicity.

3.1.3 Procedure

Participants either completed the AOT-E at the beginning of the survey or at the end (but before CRT and demographics). The presentation order did not change the pattern of results and the aggregate results will therefore be reported. Otherwise, the order of the following measures was randomized for each participant (unlike Study 1, which used a fixed order): 1) conspiracist ideation, 2) paranormal belief, 3) moral values, 4) science beliefs, 5) political opinions and free market ideology, and 6) religious belief and God skepticism. This block of questionnaires was followed by the CRT and, finally, demographics.

3.2 Results and Discussion

As is evident from Table 10 – and again supportive of the “reasoning helps” perspective – both versions of the AOT-E scale were significantly associated with every other primary measure. However, consistent with Stanovich and Toplak (2019), the correlation between the original AOT-E and religious belief (r = .42) was more than double the size of the correlation for the revised AOT-E (r = .20). The revised scale also had decreased correlations with traditional moral values (r’s = −.37 and −.17 for original and revised, respectively) and conservative opinions (r’s = −.55 and −.36 for original and revised, respectively). Nonetheless, as mentioned, the revised AOT-E was a significant predictor in every case – and, based on the norms from Gignac and Szodorai (2016), most of the correlations were medium (r = .20) to large (r = .30). Moreover, both AOT-E scales were generally more strongly correlated with the measures of interest than was CRT performance. Indeed, every measure was significantly correlated with the revised AOT-E after controlling for CRT performance (all r_partial’s > .16, all p’s < .015). Thus, it appears that one’s mere stance toward revising beliefs according to evidence may play a role in what they believe (as adults) – a conclusion that is plainly supportive of the idea that reasoning is largely effective (for some).

The pattern of correlations for the individual political opinion items was similar to Study 1 (albeit with slightly weaker effect sizes; see Table 11). Both versions of the AOT-E scale were significantly associated with liberal political stances on almost every issue, with two exceptions. The first exception, as in Study 1, was that AOT-E did not correlate with believing that microaggressions are problematic or unproblematic (in Table 11 this is coded such that a higher score indicates believing that microaggressions are unproblematic). The only notable difference between the two versions of the AOT-E (apart from the fact that the correlations tended to be stronger for the original than the revised version) was that a more strongly pro-free speech stance was nominally negatively correlated (r = −.11, p = .070) with the original AOT-E, but significantly positively correlated (r = .19, p = .002) with the revised AOT-E; this correlation was also positive using the original AOT-E in Study 1 (r = .25, p < .001; see Table 5). The latter correlation is notable because, in the Study 2 Lucid sample, conservatives more strongly disagreed that “students should be able to block controversial speakers from giving talks at their university” – a stance that was also associated with higher AOT-E (see also De keersmaecker, Bostyn, Hiel & Roets, 2020, for related results); this correlation in the same direction in the Study 1 MTurk sample (−.08, Table 2) but was not significant. In other words, the campus free speech item is the only case where higher AOT-E is associated with a stance (favoring free speech) that is positively (although modestly) correlated with conservative political ideology (Table 8). All other issues were in the opposite direction (or non-significant, as is the case for the microaggressions item).

Finally, as with the overall measures, CRT was a weaker (and often non-significant) predictor for every item relative to either AOT-E scale. Combined with Study 1, these results indicate that a major consequence of AOT-E is for political ideology – precisely the domain where motivated reasoning is purported to dominate (but for a more direct test, see Study 3).

The results for the science beliefs questionnaire largely replicated Study 1 (Table 12). That is, every science belief item that was included in both studies – including general trust in scientists – was positively correlated with both versions of the AOT-E (with the exception of the modern medicine item, which was only marginally correlated with the original AOT-E in Study 2, r = .12, p = .060). The results for the new items that were added to Study 2 were more tepid. Although disbelief in the “detoxing the body of chemicals” item was correlated with AOT-E, this was not true for any of the other new items. If anything, having a positive stance on nuclear power (the only item positively correlated, however modestly, with political conservatism; see Table 9) was nominally (but not significantly) negatively associated with the revised AOT-E (r = −.11, p = .087). Nonetheless, 10 out of 13 items (along with general trust in scientists) were correlated with the revised AOT-E in the expected direction (see also; McPhetres & Pennycook, 2020). Thus, the results again support the contention that reasoning (on balance) facilitates pro-science judgment.

4 Study 3

The results of Study 2 largely reinforced what we found in Study 1: Believing that beliefs (or opinions) should change according to evidence was associated with skepticism about conspiratorial, paranormal, and religious claims. Consistent with Stanovich and Toplak (2019), asking about opinions (revised AOT-E) instead of beliefs (original AOT-E) decreased (but did not wholly undermine) the correlation with religious belief – nonetheless, the revised AOT-E continued to significantly predict religious belief. Moreover, the revised AOT-E was just as successful at predicting conspiratorial and paranormal beliefs as the original AOT-E. Furthermore, as in Study 1, AOT-E was positively associated with care/fairness moral values and negatively associated with traditional moral values. Both versions of the AOT-E were also negatively correlated with political conservatism; including political ideology, free market ideology, and a wide range of conservative political opinions. The only exception was that the revised AOT-E was positively associated with support for campus free speech. Although this item was only modestly associated with political conservatism (r = .09), it is noteworthy that this is the sole issue out of the ten surveyed where the more politically conservative stance was associated with the stance that beliefs should change according to evidence (see also De keersmaecker et al., 2020). Both versions of the AOT-E were also predictive of a number of pro-science beliefs (with a few exceptions) (McPhetres & Pennycook, 2020). Overall, these results indicate that the AOT-E scale maintains strong predictive validity even if “opinions” are referenced instead of “beliefs”.

Although Study 2 paints a fairly clear picture in the aggregate, it remains unclear if AOT-E is predictive of (in particular) liberal opinions and pro-science beliefs across the political spectrum. Indeed, previous research has shown that cognitive sophistication interacts with political ideology when predicting people’s stance on issues such as global warming (Kahan et al., 2012; Kahan, Peters, Dawson & Slovic, 2017; Sarathchandra, Navin, Largent & McCright, 2018). Unfortunately, because we assigned participants to two different AOT-E scales, we did not have enough power in Study 2 to effectively estimate effect sizes when separating Democrats and Republicans.² Given that the original “belief” version of the AOT-E may modestly inflate some estimates of the correlation between AOT-E and a variety of issues (particularly those that have some association with religious belief), we ran a third study employing only the revised “opinion” version of the AOT-E.

4.1 Method

4.1.1 Participants

American participants were recruited from Lucid for Academics on May 9^th, 2019. We recruited 1000 participants. In total, 1063 participants began the study but 103 did not finish. We also removed 182 individuals who responded affirmatively when asked if they responded randomly at any point during the survey. The resulting sample (N = 778, Mean age = 43.8) consisted of 363 males and 410 females, 2 transgender males, 2 trans/non-binary, and 1 “not listed”.

4.1.2 Materials and Procedure

The materials and procedure were identical to Study 2, with the following exceptions: 1) Participants were administered only the revised (“opinion”) AOT-E; 2) the religious belief questionnaire (for which there was substantial missing data in Study 2) was changed from a matrix presentation format to the single-question format used for other measures; 3) we also changed the response options for the religious belief questionnaire to be consistent with the paranormal/political/science questionnaires (i.e., a 7-point scale); 4) we added 3 CRT items from Thomson and Oppenheimer (2016) that are relatively easier, based on past research (see https://osf.io/xqzse/ for full materials); and 5) we added a single continuous measure of Democrat-Republican preference (“Which of the following best describes your political preference?” Strongly Democratic, Democratic, Lean Democratic, Lean Republican, Republican, Strongly Republican), in addition to the party classification item used in Study 2 (which included “independent” as an option).

Scale reliabilities for Democrats, Republicans, and Independents are in Table 13. Reliability was low for the free market ideology scale and (among Republicans and Independents in particular) the conservative opinion and pro-science belief scales. Notably, variability was fairly similar across the major variables for Democrats and Republicans, indicating that restriction of range is not a likely explanation for any divergences that we observe.

4.2 Results and Discussion

Our focus for Study 3 was on the extent to which AOT-E predicted the same constellation of beliefs, values, and opinion for individuals across the political spectrum.⁴ As is evident from Table 14, AOT-E was a strong predictor across the board for Democrats (paralleling the overall results from Studies 1 and 2). However, the same was not equally true for Republicans, for whom AOT-E was a significant predictor of skepticism about paranormal claims, acceptance of care/fairness as moral values, and (notably) pro-science beliefs, but no other issues. Interestingly, in contrast with the overall results, AOT-E was positively associated with economic conservatism among Republicans. Nonetheless, it is noteworthy that Republicans scored themselves lower on AOT-E (M = 61.2, SD = 15) than both Democrats (M = 65.1, SD = 15.9) and Independents (M = 64.9, SD = 16.3), t’s > 2.5, p’s < .015. Moreover, the correlations when considering the full sample (averaging across liberals and conservatives) paralleled the previous two studies: AOT-E was a significant predictor for every measure except economic conservatism. Furthermore, as in Study 2, the CRT results tended to parallel AOT-E despite being a weaker predictor overall.

Turning now to specific political issues that formed our conservative opinions scale (Table 15), it is evident that the previously identified correlation between AOT-E and liberal political opinions (with one notable exception) was driven largely by Democrats and, in some cases, by Independents. For example, those higher in AOT-E were supportive of same-sex marriage among Democrats (r = −.31) and Independents (r = −.27), but this correlation is marginally significant among Republicans (r = −.12, p = .075). Most importantly, however, there was only a single issue where AOT-E predicted opposite opinions for Democrats and Republicans: Capital punishment. Whereas higher AOT-E was associated with more opposition to capital punishment among Democrats, it was associated with more support for capital punishment among Republicans. Thus, even though AOT-E was not as strongly predictive among Republicans as Democrats, it is clear that it is not merely driving political polarization either (lest more issues would be significantly correlated in opposite directions). Indeed, the opinion that there is room for men in feminism was associated with higher AOT-E for both Democrats and Republicans. Finally, the previously noted exceptional case where the more conservative opinion was, overall, associated with higher AOT-E – support for free speech – was driven by Republicans and Independents (i.e., AOT-E was not associated with support for free speech among Democrats). Thus, whether AOT-E predicts support or opposition for a specific issue appears to depend to some extent on what the issue is and about whom you’re speaking. Nonetheless, AOT-E certainly maintained a great deal of predictive validity (contrary to the “reasoning is helpless” perspective and consistent with the “reasoning is helpful” perspective) and was not associated with political polarization writ large (underming the “reasoning hurts” perspective).

The results for science-related beliefs (Table 16) parallel the pattern for conservative opinions insofar as they provide evidence against the motivated reasoning (“reasoning hurts”) perspective. Specifically, AOT-E was generally associated with pro-science beliefs across the board for Democrats (with the exceptions of skepticism about detoxing and the role of genetics in success, which were not significant, and supporting nuclear power, which was negatively associated with AOT-E) and Independents (with the exceptions of genetics and supporting GMO’s). Among Republicans, the most politically polarizing issues, such as global warming⁵, big bang, and evolution (see Table 17), were not associated with AOT-E. However, AOT-E was associated with pro-science stances on several intermediate issues; namely support for stem cells, vaccines, technology, and modern medicine. Furthermore, trust in scientists was positively associated with AOT-E across the political spectrum.

5 General Discussion

Although the belief that beliefs (and opinions) ought to change according to evidence is held by most people, there is meaningful variability in the strength of this conviction. The results of all three studies point to one broad, yet important, conclusion: Actively open-minded thinking about evidence (AOT-E) is, in the aggregate, a strong predictor of a wide range of beliefs, opinions, and values. This implies that individual differences in the propensity to reflect about evidence is something that people meaningfully engage in their everyday lives, which indicates that the exercise of human reasoning is, on balance, “helpful”. The respective ideas that reasoning is “helpless” or “hurtful” did not find support in our data.

To summarize, AOT-E was associated with skepticism about conspiratorial, paranormal, and religious claims and agreement with a variety of scientific claims. Thus, AOT-E appears to support the rejection of epistemically suspect beliefs, thereby indicating that what people believe about whether beliefs ought to change (meta-beliefs) has an influence on what they take to be true or false about the world. AOT-E was also consistently associated with political liberalism in a variety of forms (despite being a domain for which motivated reasoning should be prominent); from having a more liberal political ideology, to adoption of more liberal moral values (specifically, rejection of traditional values and agreement with care/fairness values), to opposition to economic conservatism and free-market ideology. Furthermore, AOT-E was positively associated with a variety of specific liberal political opinions (e.g., supporting gay marriage and access to abortion) in the aggregate. This suggests that political conservatives, who tend to be more resistant to societal change (White, Kinney, Danek, Smith & Harben, 2019), may also be more resistant to intrapersonal belief change (but see below for a more nuanced interpretation).

The strong predictive validity of the AOT-E across a wide range of domains suggests that people’s meta-beliefs about whether and how beliefs should change play an important role in belief formation. However, this conclusion comes with an important caveat that is in many ways as interesting as the conclusion itself. Most notably, Study 3 revealed that AOT-E is much more predictive among Democrats than among Republicans (with Independents being intermediate).⁶ This interaction is, in some cases, consistent with previous research that has been used to support the “reasoning hurts” perspective – however, as we will argue, it is not consistent with how some of these past results have been interpreted.

5.1 AOT-E among Democrats and Republicans

The AOT-E did not have the same predictive validity for conservatives as it did for liberals. To take a prominent example from Study 3, AOT-E was very strongly correlated with belief in anthropogenic climate change among Democrats (r = .43, p < .001), but there was no such (significant) correlation among Republicans (r = .09, p = .179). This parallels previous findings wherein individuals who are more cognitively sophisticated (using a variety of measures, including the CRT) are more politically polarized in terms of climate change (Kahan et al., 2012). In particular, Kahan et al. found that science literacy and numeracy was positively correlated with climate change risk attitudes among liberals (r = .08, p = .03) but negatively correlated among conservatives (r = −.12, p = .03).⁷ The favored explanation for these results is that cognitive sophistication polarizes climate change (and other) attitudes because it facilitates motivated (“identity-protective”) reasoning (Kahan et al., 2012; Kahan, Peters, Dawson & Slovic, 2017; Sarathchandra, Navin, Largent & McCright, 2018) – an account that has notably been applied to the AOT as a measure of general cognitive sophistication as well (Kahan & Corbin, 2016; but see Baron, 2017). This “humans-as-lawyers” motivated reasoning perspective has had a large influence on the field and in the popular press (for an overview, see Pennycook, 2018).

Motivated reasoning cannot, however, account for our broad pattern of results (with some potential exceptions). Although an interaction between political ideology and cognitive sophistication in the prediction of an attitude (such as climate change belief) is often taken as positive evidence for the motivated reasoning account, the interaction is easily understood as a consequence of the sample characteristics. For example, the interaction between political ideology and cognitive sophistication in the prediction of climate change attitudes emerges because the sample happens to have both liberals and conservatives. However, an interaction can emerge from opposing effects that are not individually significant. This is important because the central prediction of the motivated reasoning (“reasoning hurts”) account is actually two separate (and opposing) main effects for political liberals and conservatives: Cognitive sophistication should be positively associated with politically congruent attitudes (e.g., climate change for liberals) and negatively associated with politically incongruent attitudes (e.g., climate change for conservatives; see Pennycook & Rand, 2019b). The results of Study 3 are plainly inconsistent with this prediction: Not only did we not find opposing main effects in the context of climate change (in fact, AOT-E is nominally positively correlated with climate change beliefs among Republicans), we do not find it for any other polarized issue either (with one exception). Specifically, there was not a single scientific issue that we included in our study that produced significant correlations with AOT-E in opposite directions for Democrats and Republicans. Furthermore, across ten explicitly political issues (such as support for police authority or opposition to abortion) there was only one case where the motivated reasoning prediction of significant opposing effects was present: Capital punishment. Even broad ideological positions such as social conservatism and free-market ideology did not produce significant opposing effects (although a second exception is present here: Economic conservatism). Thus, in almost every case, the motivated reasoning (or identity-protective cognition) account’s prediction was not supported. Rather, it appears that AOT-E is simply less predictive among political conservatives than among liberals.

One potential explanation of this is that, despite arguments to the contrary (Kahan & Corbin, 2016), actively open-minded thinking about evidence is not merely a proxy for cognitive sophistication (Baron et al., 2015) – an observation supported by the divergences between AOT-E and CRT in our own data (with the former being a stronger predictor overall than the latter) as well as the data of Kahan and Corbin (Baron, 2017). Indeed, as intimated in the introduction, AOT-E is definitionally opposed to motivated reasoning: Believing that beliefs ought to change according to evidence essentially amounts to a rejection of motivated reasoning. Of course, it is possible that those who report being more actively open-minded are simply being deceptive (and potentially self-deceptive). That is, individuals who say that they are particularly willing to change their beliefs according to evidence are, in reality, the most likely to do the opposite and engage in motivated reasoning. This seems unlikely. And, at any rate, the results for the CRT – which is plainly a measure of some sort of cognitive sophistication – also do not support the motivated reasoning account. There was not a single specific issue, political or science-based, that produced opposing and significant correlations with CRT for Democrats relative to Republicans. Thus, the most parsimonious take-away from the present data is simply that the motivated reasoning account (a “reasoning hurts” perspective) is wrong or incomplete.

If not motivated reasoning, what then explains the finding that AOT-E is more consistently predictive for liberals than conservatives? It is potentially revealing that the depression in predictive validity for Republicans relative to Democrats was seemingly evident even for measures that did not significantly correlate with conservatism. For example, conspiracy ideation was strongly correlated with AOT-E for Democrats (r = −.32, p < .001) but less so for Republicans (although it was marginally significant, r = −.12, p = .070).⁸ This occurred even though conspiracy belief was equivalent between Democrats and Republicans (t = 1.16, p = .245). One mundane possibility is that data quality was, for whatever reason, poorer among Republicans than Democrats. Contrary to this, scale reliabilities were largely similar for both groups (Table 13). Moreover, random responding was very similar for Republicans and Democrats.

One possibility is that there are important differences between the “coalitions” that make up the Democratic and Republican parties. For example, Baron (2017) noted that the Democratic Party in the United States (and liberals in general) is made up of a more diverse group of people than is the Republican Party. Supporting this idea, variation in most of the primary measures in Studies 2 and 3 (including the AOT-E itself) is at least nominally higher among Democrats than Republicans (see Table S4 in the supplementary materials).

Yet another possibility is that there is no genuine difference between conservatives and liberals in terms of people’s beliefs about how beliefs should change, but the AOT-E items are viewed through a political lens in the similar sort of way that the “belief” items in the original AOT-E were biased against religious individuals (Stanovich & Toplak, 2019). It may be that “evidence” as a term or concept has been politicized to some extent and that AOT-E may be viewed as a liberal outlook (Krugman, 2019). Under this account, the weaker correlations among Republicans occurs because some proportion of conservatives are reporting lower AOT-E simply because they are resisting the framing or wording of the questions (or perhaps the source of the questions – for more on insincere responding in the context of partisan bias in surveys, see Bullock & Lenz, 2019). One counter to this possibility, however, is that the CRT is also less predictive among Republicans. Republicans did no worse on the test than Democrats and presumably are not ideologically opposed to simple-seeming word problems. Although this does not rule out the politicization of evidence possibility, it does render it less likely.

Alternatively, liberals and conservatives (in the USA) may genuinely differ not only in what they believe (including meta-beliefs), but why they believe it. The AOT-E is equipped to assess one’s stance toward evidence, which is apparently important among liberals (insofar as AOT-E distinguishes between what types of beliefs and opinions liberals tend to have – although, of course, other factors are surely important as well). At least based on the present correlational data, belief formation appears to be driven more by other factors for conservatives. That is, it is not simply that conservatives are less willing to change their beliefs according to evidence (although the overall difference is nonetheless evident), but rather that factors unstudied here contribute more substantially to belief formation among conservatives. One of the apparent defining features of conservatism, apart from resistance to change, is the endorsement of hierarchies (e.g., Jost, Glaser, Kruglanski & Sulloway, 2003). Perhaps part of the reason why AOT-E is less predictive among conservatives, then, is that beliefs are less intrapersonal and more interpersonal among political conservatives. That is, belief is more about social groups and, therefore, variation in beliefs among conservatives is driven more by exposure to different hierarchies and information sources (for an example, see Landrum, Lull, Akin, Hasell & Jamieson, 2017). Plainly, further research is required.

5.2 Very large effect sizes: A lesson

In a recent discussion of effect size estimates, Funder and Ozer (2019) argued that r’s of .10, .20, and .30 correspond to small, medium, and large effect sizes, respectively (see also Gignac & Szodorai, 2016). They also argued that very large effect sizes (r = .40 or greater) are, in the context of psychological research, “likely to be a gross overestimates rarely found in a large sample or in a replication” (p. 1). In Study 1, AOT-E predicted multiple beliefs and opinions at a level greater than .40 (including aggregate liberal opinions and pro-science beliefs at r’s = ~.60). As a meaningful counter-example to Funder and Ozer, Study 1 consisted of a large sample (N = 375) and was largely replicated with a different sample (using the original AOT-E) in Study 2 (r’s were .55 and .40 for liberal opinions and pro-science beliefs, respectively). Nonetheless, consistent with Funder and Ozer’s larger point, the very large effect sizes in Study 1 may be inflated for two reasons.

First, as argued by Stanovich and Toplak (2019) (who also noted the large effect sizes as a reason for skepticism), the original version of the AOT-E appears to have inflated some effect sizes because individuals may have presumed the questions to be about religious belief in particular instead of beliefs more generally. Although religious believers continue to rate themselves as more resistant to revising opinions according to evidence relative to non-believers, the “belief” wording in the original AOT-E (which was derived from earlier scales) may have inflated the correlation with religious belief and its covariates. However, one alternative possibility is that the belief items are simply more predictive overall (e.g., because they are more easily understood). A more systematic investigation of “belief” versus “opinion” wording is necessary to come to firm conclusions. Of course, the present data indicate that the AOT-E is relatively strongly predictive regardless of these small changes in wording.

Second, as discussed, we found that AOT-E was much more predictive across the board for political liberals (Democrats) than for conservatives (Republicans). This is relevant for the apparently over-estimated correlation effect sizes in Study 1 because that sample came from Mechanical Turk, which was heavily liberal-skewed.⁹ Thus, although Study 1 consisted of a large sample and produced results that were replicated in Study 2, our evidence indicates that Funder & Ozer’s (2019) conclusion about very large effect sizes being likely overestimates is nonetheless accurate. However, in this case, the issue was more a matter of generalizability than replicability. The underlying lesson is the same: Very large effect sizes should be interpreted with caution.

5.3 Limitations

The principal limitation of the present study is that it is correlational and therefore not possible to establish, for example, whether AOT-E affects political opinions, vice versa, and/or some third factor affects both. Nonetheless, experimentally manipulating AOT-E and testing for a change in beliefs does not seem a prudent approach. Beliefs, opinions, and values are formed across years, and minute-long manipulations do not offer a reasonable proxy for the psychological processes that are of chief interest here. Rather, longitudinal studies that establish differences in AOT-E at adolescence and test for changes in beliefs over time would be a more fruitful future direction.

The generalizability of this study is limited in a number of ways. First, our samples are not precisely representative of the United States population (although Lucid is much closer than Mechanical Turk). Second, we obviously cannot generalize beyond the USA. Third, although we attempted to test as many different beliefs, values, and opinions as possible in a single survey, we have surely missed many important issues. Furthermore, it is possible that our own selection of issues was subject to our own liberal political bias.

6 Conclusion

Our 8-item actively open-minded thinking about evidence (AOT-E) scale was strongly predictive of a wide range of beliefs, values, and opinions. People who reported believing that beliefs and opinions should change according to evidence were less likely to be religious, less likely to hold paranormal and conspiratorial beliefs, more likely to believe in a variety of scientific claims, and were more political liberal (in terms of overall ideology, partisan affiliation, moral values, and a variety of specific political opinions). Moreover, the effect sizes for these correlations was often large or very large, based on established norms (Funder & Ozer, 2019; Gignac & Szodorai, 2016). The size and diversity of AOT-E correlates strongly supports one major, if broad, conclusion: Socio-cognitive theories of belief (both specific and general) should take into account what people believe about when and how beliefs and opinions should change (i.e., meta-beliefs). That is, we should not assume that evidence is equally important for everyone. Furthermore, our findings provide clear support for the perspective that reasoning facilitates sound judgment, thereby undermining the idea that intuition commonly dominates reasoning. We also found essentially no support for motivated reasoning. Regardless future work is required to more clearly delineate why AOT-E is more predictive for political liberals than conservatives.

References

Bargh, J. A. (1999). The cognitive monster: The case against the controllability of automatic stereotype effects. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology (pp. 361–382). Guilford Press.

Bargh, J. A., & Chartrand, T. L. (1999). The unbearable automaticity of being. American Psychologist, 54(7), 462–479. https://doi.org/10.1037/0003-066X.54.7.462.

Bargh, J. A., Schwader, K. L., Hailey, S. E., Dyer, R. L., & Boothby, E. J. (2012). Automaticity in social-cognitive processes. Trends in Cognitive Sciences, 16(12), 593–605. https://doi.org/10.1016/j.tics.2012.10.002.

Baron, J. (1985). What kinds of intelligence components are fundamental? In S. F. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills: Vol. II, Research and open questions. Hillsdale, NJ: Lawrence Earlbaum.

Baron, J. (2008). Thinking and deciding (4th ed.). Cambridge: Cambridge University Press.

Baron, J. (2017). Comment on Kahan and Corbin: Can polarization increase with actively open-minded thinking? Research & Politics, 4(1), 1–4. https://doi.org/10.1177/2053168016688122.

Baron, J. (2019). Actively open-minded thinking in politics. Cognition. https://doi.org/10.1016/j.cognition.2018.10.004.

Baron, J., & Jost, J. T. (2019). False equivalence: Are liberals and conservatives in the United States equally biased? Perspectives on Psychological Science, 14(2), 292–303. https://doi.org/10.1177/1745691618788876.

Baron, J., Scott, S., Fincher, K., & Emlen Metz, S. (2015). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 4(3), 265–284. https://doi.org/10.1016/j.jarmac.2014.09.003.

Brotherton, R., & French, C. (2013). Measuring belief in conspiracy theories: The generic conspiracist beliefs scale. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2013.00279.

Bullock, J. G., & Lenz, G. (2019). Partisan bias in surveys. Annual Review of Political Science, 22(1), 325–342. https://doi.org/10.1146/annurev-polisci-051117-050904.

Campitelli, G., & Gerrans, P. (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42(3), 434–447. https://doi.org/10.3758/s13421-013-0367-9.

Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40–48. https://doi.org/10.1177/1745691613513470.

Cohen, P., Cohen, J., Aiken, L. S., & West, S. G. (1999). The problem of units and the circumstance for POMP. Multivariate Behavioral Research, 34, 315–346. https://doi.org/10.1207/S15327906MBR3403\_2.

Coppock, A., & Mcclellan, O. A. (2019). Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents. Research & Politics. Retrieved from https://alexandercoppock.com/papers/CM\_lucid.pdf.

De keersmaecker, J., Bostyn, D. H., Hiel, A. Van, & Roets, A. (2020). Disliked but free to speak: Cognitive ability is related to supporting freedom of speech for groups across the ideological spectrum. PsyArXiv, 1–21. https://doi.org/https://doi.org/10.31234/OSF.IO/B7KTY

De Neys, W. (2017). Dual process theory 2.0. London, UK: Routledge.

Dijksterhuis, A., & Strick, M. (2016). A case for thinking without consciousness. Perspectives on Psychological Science, 11(1), 117–132. https://doi.org/10.1177/1745691615615317.

Ditto, P. H., Liu, B. S., Clark, C. J., Wojcik, S. P., Chen, E. E., Grady, R. H., & Zinger, J. F. (2019). At least bias is bipartisan: A meta-analytic comparison of partisan bias in liberals and conservatives. Perspectives on Psychological Science, 14, 273–291. https://doi.org/10.1007/s10551-015-2769-z.For.

Evans, J. S. B. T., & Stanovich, K. E. (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3), 223–241. https://doi.org/10.1177/1745691612460685.

Evans, J. St. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278. https://doi.org/10.1146/annurev.psych.59.103006.093629.

Evans, J. St. B. T. (2012). Dual process theories of deductive reasoning: facts and fallacies. In The Oxford handbook of thinking and reasoning (pp. 115–133). New York: Oxford U.

Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42. https://doi.org/10.1257/089533005775196732.

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 251524591984720. https://doi.org/10.1177/2515245919847202.

Gervais, W. M. (2015). Override the controversy: Analytic thinking predicts endorsement of evolution. Cognition, 142, 312–321. https://doi.org/10.1016/j.cognition.2015.05.011.

Gigerenzer, G. (2007). Gut feelings: The intelligence of the unconscious. Penguin Books.

Gigerenzer, G., Todd, P., & Group, the A. R. (1999). Simple heuristics that make us smart. Oxford, England: Oxford University Press.

Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069.

Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & Ditto, P. H. (2011). Mapping the moral domain. Journal of Personality and Social Psychology, 101(2), 366–385. https://doi.org/10.1037/a0021847.

Haidt, J. (2012). The righteous mind: Why good people are divided by politics and religion. New York, NY: Paragon.

Haidt, Jonathan. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4), 814–834. https://doi.org/10.1037//0033-295X.

Haran, U., Ritov, I., & Mellers, B. a. (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment and Decision Making, 8(3), 188–201. https://doi.org/10.1017/CBO9781107415324.004.

Heath, Y., & Gifford, R. (2006). Free-market ideology and environmental degradation the case of belief in global climate change. Environment and Behavior, 38, 48–71. https://doi.org/10.1177/0013916505277998.

Jost, J. T. (2017). Ideological asymmetries and the essence of political psychology. Political Psychology, 38(2), 167–208. https://doi.org/10.1111/pops.12407.

Jost, J. T., Glaser, J., Kruglanski, A. W., & Sulloway, F. J. (2003). Political conservatism as motivated social cognition. Psychological Bulletin, 129(3), 339–375. https://doi.org/10.1037/0033-2909.129.3.339.

Kahan, D. M. (2013). Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making, 8(4), 407–424. https://doi.org/10.2139/ssrn.2182588.

Kahan, D. M., & Corbin, J. C. (2016). A note on the perverse effects of actively open-minded thinking on climate-change polarization. Research and Politics, 1–5. https://doi.org/10.1177/2053168016676705.

Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L. L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change, 2(10), 732–735. https://doi.org/10.1038/nclimate1547.

Kahan, D., Peters, E., Dawson, E., & Slovic, P. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, 1(1), 54–86.

Kahneman, D, Slovic, P., & Tversky, A. (1982). Judgments under uncertainty: Heuristics and biases. Cambridge, MA: Cambridge University Press.

Kahneman, Daniel, & Klein, G. (2009). Conditions for intuitive expertise: a failure to disagree. The American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755.

Klein, G. (2008). Naturalistic decision making. Human Factors, 50(3), 456–460. https://doi.org/10.1518/001872008X288385.

Knobloch-Westerwick, S., Mothes, C., & Polavin, N. (2017). Confirmation bias, ingroup bias, and negativity bias in selective exposure to political information. Communication Research, 1–21. https://doi.org/10.1177/0093650217719596.

Kohlberg, L. (1969). Stage and sequence: The cognitive-developmental approach to socialization. In D. A. Goslin (Ed.), Handbook of socialization theory and research (pp. 347–480).

Krugman, P. (2019, April). Republicans are the real extremists. The New York Times. Retrieved from https://www.nytimes.com/2019/04/15/opinion/republicans-ocasio-cortez-omar.html.

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498. https://doi.org/10.1037/0033-2909.108.3.480.

Landrum, A. R., Lull, R. B., Akin, H., Hasell, A., & Jamieson, K. H. (2017). Processing the papal encyclical through perceptual filters: Pope Francis, identity-protective cognition, and climate change concern. Cognition, 166, 1–12. https://doi.org/10.1016/j.cognition.2017.05.015.

Lodge, M., & Taber, C. (2005). The automaticity of affect for political candidates, parties and issues. Political Psychology, 26(3), 455–482.

McPhetres, J., & Pennycook, G. (2020). Science beliefs, political ideology, and cognitive sophistication. PsyArXiv Working Paper, 1–33. https://doi.org/10.31219/OSF.IO/AD9V7.

Mercier, H. (2016). The argumentative theory: Predictions and empirical evidence. Trends in Cognitive Sciences, 20(9), 689–700. https://doi.org/10.1016/j.tics.2016.07.001.

Mercier, H., & Sperber, D. (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2), 57–74. https://doi.org/10.1017/S0140525X10000968.

Metz, S. E., Weisberg, D. S., & Weisberg, M. (2018). Non-Scientific criteria for belief sustain counter-scientific beliefs. Cognitive Science, 42(5), 1477–1503. https://doi.org/10.1111/cogs.12584.

Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175.

Pennycook, G. (2018). The new reflectionism in cognitive psychology: Why reason matters. New York, NY: Routledge.

Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2014a). Cognitive style and religiosity: the role of conflict detection. Memory & Cognition, 42(1), 1–10. https://doi.org/10.3758/s13421-013-0340-7.

Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2014b). The role of analytic thinking in moral judgements and values. Thinking & Reasoning, 20(2), 188–214. https://doi.org/10.1080/13546783.2013.865000.

Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2015). On the reception and detection of pseudo-profound bullshit. Judgment and Decision Making, 10(6), 549–563.

Pennycook, G., Cheyne, J. A., Koehler, D. J., & Fugelsang, J. A. (2016). Is the cognitive reflection test a measure of both reflection and intuition? Behavior Research Methods, 48, 341–348. https://doi.org/10.3758/s13428-015-0576-1.

Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (2012). Analytic cognitive style predicts religious and paranormal belief. Cognition, 123(3), 335–346. https://doi.org/10.1016/j.cognition.2012.03.003.

Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015a). Everyday consequences of analytic thinking. Current Directions in Psychological Science, 24(6), 425–432. https://doi.org/10.1177/0963721415604610.

Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015b). What makes us think? A three-stage dual-process model of analytic engagement. Cognitive Psychology, 80, 34–72. https://doi.org/10.1016/j.cogpsych.2015.05.001.

Pennycook, G., & Rand, D. G. (2019a). Cognitive reflection and the 2016 US presidential election. Personality and Social Psychology Bulletin, 45, 224–239. https://doi.org/10.1177/0146167218783192.

Pennycook, G., & Rand, D. G. (2019b). Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition, 188, 39–50. https://doi.org/10.1016/j.cognition.2018.06.011.

Pennycook, G., Ross, R. M., Koehler, D. J., & Fugelsang, J. A. (2016). Atheists and agnostics are more reflective than religious believers: Four empirical studies and a meta-analysis. Plos One, 11(4), e0153039. https://doi.org/10.1371/journal.pone.0153039.

Perkins, D. (2019). Learning to reason: The influence of instruction, prompts and scaffolding, metacognitive knowledge, and general intelligence on informal reasoning about everyday social and political issues. Judgment and Decision Making, 14(6), 624–643. Retrieved from http://journal.sjdm.org/19/190925a/jdm190925a.pdf.

Piaget, J. (1932). The moral judgment of the child. London, UK: Routledge Kegan Paul.

Price, E., Ottati, V., Wilson, C., & Kim, S. (2015). Open-minded cognition. Personality and Social Psychology Bulletin, 41(11), 1488–1504. https://doi.org/10.1177/0146167215600528.

Pronin, E., Lin, D. Y., & Ross, L. (2002). The bias blind spot: Perceptions of bias in self versus others. Personality and Social Psychology Bulletin, 28(3), 369–381. https://doi.org/10.1177/0146167202286008.

Redlawsk, D. (2002). Hot cognition or cool consideration? Testing the effects of motivated reasoning on political decision making. Journal of Politics, 64, 1021–1044. https://doi.org/10.1111/1468-2508.00161.

Redlawsk, D. P., Civettini, A. J. W., & Emmerson, K. M. (2010). The affective tipping point: Do motivated reasoners ever “get it”? Political Psychology, 31(4), 563–593. https://doi.org/10.1111/j.1467-9221.2010.00772.x.

Ross, L., & Ward, A. (1996). Naive realism in everyday life: Implications for social conflict and misunderstanding. In E. S. Reed, E. Turiel, & T. Brown (Eds.), The Jean Piaget symposium series. Values and knowledge (pp. 103–135). Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc.

Royzman, E. B., Landy, J. F., & Goodwin, G. P. (2014). Are good reasoners more incest-friendly[202F?]? Trait cognitive reflection predicts selective moralization in a sample of American adults. Judgment and Decision Making, 9(3), 176–190.

Rutjens, B. T., Sutton, R. M., & van der Lee, R. (2018). Not all skepticism is equal: Exploring the ideological antecedents of science acceptance and rejection. Personality and Social Psychology Bulletin, 44(3), 384–405. https://doi.org/10.1177/0146167217741314.

Sá, W. C., West, R. F., & Stanovich, K. E. (1999). The domain specificity and generality of belief bias: Searching for a generalizable critical thinking skill. Journal of Educational Psychology, 91(3), 497–510. https://doi.org/10.1037/0022-0663.91.3.497.

Sarathchandra, D., Navin, M. C., Largent, M. A., & McCright, A. M. (2018). A survey instrument for measuring vaccine acceptance. Preventive Medicine, 109, 1–7. https://doi.org/10.1016/J.YPMED.2018.01.006.

Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology. General, 141(3), 423–428. https://doi.org/10.1037/a0025391.

Stanovich, K. E. (2005). The robot’s rebellion: Finding meaning in the age of Darwin. Chicago, IL: Chicago University Press.

Stanovich, K. E., & Toplak, M. E. (2019). The need for intellectual diversity in psychological science: Our own studies of actively open-minded thinking as a case study. Cognition, 187, 156–166. https://doi.org/10.1016/j.cognition.2019.03.006.

Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89(2), 342–357. https://doi.org/10.1037/0022-0663.89.2.342.

Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal of Experimental Psychology: General, 127(2), 161–188. https://doi.org/10.1037/0096-3445.127.2.161.

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645–665; discussion 665-726. https://doi.org/10.1017/S0140525X00003435.

Stanovich, K. E., West, R. F., & Toplak, M. E. (2013). Myside bias, rational thinking, and intelligence. Current Directions in Psychological Science, 22(4), 259–264. https://doi.org/10.1177/0963721413480174.

Strickland, A. A., Taber, C. S., & Lodge, M. (2011). Motivated Reasoning and Public Opinion. Journal of Health Politics, Policy and Law, 36(6), 89–122. https://doi.org/10.1215/03616878-

Svedholm-Häkkinen, A. M., & Lindeman, M. (2018). Actively open-minded thinking: development of a shortened scale and disentangling attitudes towards knowledge and people. Thinking and Reasoning, 24, 21–40. https://doi.org/10.1080/13546783.2017.1378723.

Svedholm, A. M., & Lindeman, M. (2013). The separate roles of the reflective mind and involuntary inhibitory control in gatekeeping paranormal beliefs and the underlying intuitive confusions. British Journal of Psychology, 104(3), 303–319. https://doi.org/10.1111/j.2044-8295.2012.02118.x.

Swami, V., Voracek, M., Stieger, S., Tran, U. S., & Furnham, A. (2014). Analytic thinking reduces belief in conspiracy theories. Cognition, 133(3), 572–585. https://doi.org/10.1016/j.cognition.2014.08.006.

Thompson, V. A., Prowse Turner, J. A., & Pennycook, G. (2011). Intuition, reason, and metacognition. Cognitive Psychology, 63(3), 107–140. https://doi.org/10.1016/j.cogpsych.2011.06.001.

Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99–113.

Tobacyk, J. J. (2004). A revised paranormal belief scale. The International Journal of Transpersonal Studies, 23(3), 94–98.

Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275–1289. https://doi.org/10.3758/s13421-011-0104-1.

White, K. R. G., Kinney, D., Danek, R. H., Smith, B., & Harben, C. (2019). The Resistance to Change-Beliefs Scale: Validation of a new measure of conservative ideology. Personality and Social Psychology Bulletin. https://doi.org/10.1177/0146167219841624.

Hill/Levene Schools of Business, University of Regina, Canada. Email: gordon.pennycook@uregina.ca.

Department of Psychology, University of Waterloo, Canada.

Preparation of this manuscript was supported by the Social Sciences and Humanities Research Council of Canada and the Natural Sciences and Engineering Research Council of Canada.

We subsequently realized that the wording of this item may be problematic: Individuals might indicate that microaggressions are a problem because people are being microaggressed, but others might indicate that they are a problem because people are (falsely, presumably) indicating that they are being microaggressed.

Only 97 Democrats, 88 Republicans, and 75 Independents were administered the revised AOT-E in Study 2.

Some participants (N = 14) also chose “other” for the political party question. They are not included in the Democrat/Republican/Independent groups but are included in the overall analysis.

We will focus here on party affiliation, but the results are similar if liberals and conservatives are separated using the political ideology measures or the continuous party identification measure.

Consistent with Kahan and Corbin (2016), we found a significant interaction between belief in anthropogenic global warming and political partisanship (Democrat vs. Republican) in predicting AOT-E, β = .30, p < .001. However, as shown in Table 16, the coefficient for the correlation between global warming and AOT-E among Republicans is actually positive (instead of negative), which is contrary to the claim that AOT-E is associated with more skepticism about global warming among Republicans.

Nonetheless, AOT-E was significantly correlated with every primary measure in all three studies even after political ideology and demographics were statistically controlled (see supplementary materials for full analysis). The same was true for CRT performance (albeit to a lesser extent), except for care/fairness moral values and free market ideology.

We see a more direct parallel when considering the association between CRT and belief in anthropogenic global warming in Study 3: r = .19, p = .001 among Democrats; r = −.12, p = .088 among Republicans. Furthermore, Kahan and Corbin (2016) also found a significant positive correlation between views on climate change and AOT among Democrats and a null (but nominally negative) correlation among Republicans.

However, the interaction between AOT-E and a continuous measure of political partisanship (Democrat vs. Republican) in the prediction of conspiracy beliefs was only marginally significant, β = .06, p = .093.

It is also noteworthy that reliability for the original AOT-E was greater for the Mechanical Turk sample (.87) than the Lucid sample (.72). It is possible that the muted effect sizes in Studies 2 and 3 are also the result of lower data quality on Lucid than on Mechanical Turk (and, indeed, random responding was much more common in the former than the latter). This may also explain why the CRT was broadly less predictive in the Lucid sample than in past studies. CRT scores were, overall, quite low on Lucid.

#	Item	AOT Subscale
1	A person should always consider new possibilities.	AOT
2	People should always take into consideration evidence that goes against their beliefs.	AOT
3	It is important to persevere in your beliefs even when evidence is brought to bear against them. (rev)	Belief Identification
4	Certain beliefs are just too important to abandon no matter how good a case can be made against them. (rev)	Belief Identification
5	One should disregard evidence that conflicts with your established beliefs. (rev)	Belief Identification
6	Beliefs should always be revised in response to new information or evidence.	Belief Identification
7	No one can talk me out of something I know is right. (rev)	Dogmatism
8	I believe that loyalty to one’s ideals and principles is more important than “open-mindedness”. (rev)	Openness-Values

Name	Item	Correlation (r) with conservatism	Mean (SD)
Abortion	Abortion should be legally available with few or no restrictions. (rev)	−.59	67.3 (35.1)
Same Sex Marriage	People of the same sex should be permitted to marry. (rev)	−.53	78.0 (32.7)
Military	One of the most important things a government can do is to make sure that its military forces remain strong, even if social programs have to be cut back.	.52	44.8 (31.5)
War	It is better for a country to be ready to go to war than to be pushed around or to waste time in negotiations.	.42	43.3 (30.7)
Police Authority	There is too much focus on the rights of suspects, because police need more authority to deal with criminals.	.40	35.1 (30.0)
Men in Feminism	Is there room for men in feminism? (rev)	−.39	67.7 (28.8)
Capital Punishment	It is important to have the death penalty available as a deterrent to very serious crimes such as murder.	.38	53.6 (36.3)
Sexism for Men	Men experience sexism on par with women.	.24	39.7 (30.2)
Microaggressions	Microaggressions are a serious problem in educational contexts (such as in universities)^τ. (rev)	−.19	53.4 (28.5)
Campus Free Speech	Students should be able to block controversial speakers from giving talks at their university. (rev)	−.08	33.0 (30.6)
^τThe following note was also presented: Microaggressions are defined as “brief, everyday exchanges that send denigrating messages to certain individuals because of their group membership.”
All corrlations are significant at p<.001 except the last, which is n.s.

Name	Item	Correlation (r) with conservatism	Mean (SD)
Global warming	Global warming is at least partly caused by human activity and is a serious problem for the environment.	−.55	80.6 (27.5)
Evolution	Evolution is the best explanation so far for our origins.	−.43	73.7 (31.0)
Big Bang	The big bang theory is, generally speaking, the best explanation we have so far for the origin of our universe.	−.39	70.6 (31.3)
Stem Cells	Stem cell research is a productive enterprise.	−.32	77.0 (25.2)
Vaccines/Autism	Vaccines can cause autism in children. (rev)	−.31	26.0 (29.3)
Old Earth	The universe is billions of years old.	−.29	85.5 (23.1)
Modern Medicine	Modern medicine is the most effective means of treating most disease.	−.17	78.4 (24.0)
Tech Problems	Technology causes more problems than it helps solve. (rev)	−.16	29.3 (27.7)
GMO/Health	Genetically modified foods are hazardous to human health. (rev)	−.16	51.7 (31.3)

	1	2	3	4	5	6	7	8	9	10	11
1. AOT-E	(.87)
2. Conspiracy Beliefs	-.36***	(.95)
3. Paranormal Beliefs	-.40***	.63***	(.97)
4. Traditional MV	-.54***	.16**	.27***	(.80)
5. Care/Fairness MV	.26***	-.10	-.09	.12*	(.85)
6. Social Conservatism	-.46***	.20***	.18**	.46***	-.16**	...
7. Fiscal Conservatism	-.24***	.11*	.12*	.27***	-.11*	.69***	...
8. Free Market Ideology	-.34***	.13*	.11*	.29***	-.18**	.45***	.53***	(.85)
9. Conservative Opinions	-.61***	.28***	.30***	.64***	-.23***	.72***	.54***	.55***	(.79)
10. Pro-Science Beliefs	.65***	-.52***	-.49***	-.47***	.17**	-.54***	-.32***	-.31***	-.63***	(.84)
11. God Skepticism	.52***	-.28***	-.48***	-.48***	.02	-.37***	-.24***	-.19***	-.47***	.55***	...
*indicates p < .001, indicates p < .01, *indicates p < .05.

	1	2	3	4	5	6	7	8	9	10	11
1. AOT-E
2. Abortion	-.43***
3. Same Sex Marriage	-.50***	.63***
4. Military	-.40***	.40***	.32***
5. War	-.37***	.26***	.29***	.62***
6. Police Authority	-.51***	.35***	.43***	.48***	.44***
7. Men in Feminism	-.37***	.38***	.47***	.30***	.28***	.33***
8. Capital Punishment	-.33***	.15**	.23***	.46***	.43***	.46***	.19***
9. Sexism for Men	-.24***	.12*	.10*	.30***	.23***	.21***	.22***	.22***
10. Microaggressions	.06	.04	.002	.07	-.04	-.07	.14***	.001	.17***
11. Campus Free Speech^τ	.25***	-.01	-.08	-.03	-.03	-.17**	.05	-.08	-.06	.24***
12. Trust in Government	-.23***	-.004	.03	.10	.08	.27***	.01	.09	.13*	-.05	-.22***
^τ Campus free speech was not significantly associated with conservatism (see Table 2). At any rate, a high score indicates opposition to the idea that “students should be able to block controversial speakers from giving talks at their university”.
*indicates p < .001, indicates p < .01, *indicates p < .05.

	1	2	3	4	5	6	7	8	9	10	11
1. AOT-E	−
2. Global warming	.37^***	−
3. Evolution	.51^***	.38^***	−
4. Big Bang	.51^***	.34^***	.78^***	−
5. Stem Cells	.45^***	.31^***	.55^***	.54^***	−
6. Vaccines/Autism	.47^**	.26^***	.39^***	.37^***	.40^***	−
7. Old Earth	.40^***	.33^***	.47^***	.45^***	.41^***	.38^***	−
8. Modern Medicine	.33^***	.18^***	.36^***	.37^***	.47^***	.53^***	.33^***	−
9. Tech Problems	.44^***	.15^**	.28^***	.28^***	.36^***	.49^***	.21^***	.47^***	−
10. GMO/Health	.36^***	.06	.33^***	.37^***	.34^***	.52^***	.24^***	.42^***	.39^***	−
11. Trust Scientists	.35^***	.42^***	.48^***	.49^***	.47^***	.43^***	.29^***	.43^***	.33^***	.31^***	−
* indicates p < .001, indicates p < .01, *indicates p < .05.

Original AOT-E	Revised AOT-E
A person should always consider new possibilities.	A person should always consider new information.
People should always take into consideration evidence that goes against their beliefs.	People should always take into consideration evidence that goes against their opinions.
It is important to persevere in your beliefs even when evidence is brought to bear against them. (rev)	It is important to persevere in your opinions even when evidence is brought to bear against them. (rev)
Certain beliefs are just too important to abandon no matter how good a case can be made against them. (rev)	Certain opinions are just too important to abandon no matter how good a case can be made against them. (rev)
One should disregard evidence that conflicts with your established beliefs. (rev)	One should disregard evidence that conflicts with your established opinions. (rev)
Beliefs should always be revised in response to new information or evidence.	Opinions should always be revised in response to new information or evidence.
No one can talk me out of something I know is right. (rev)	It is possible for someone to convince me to change my mind.
I believe that loyalty to one’s ideals and principles is more important than “open-mindedness”. (rev)	I believe that loyalty to one’s ideals and principles is more important than “open-mindedness”. (rev)

	Democrats	Republicans	Independents
AOT-E (revised)	.72	.71	.73
CRT	.69	.73	.64
Conspiracy Beliefs	.94	.94	.93
Paranormal Beliefs	.96	.95	.94
Traditional MV	.75	.71	.74
Care/Fairness MV	.89	.85	.89
Free Market Ideology	.53	.64	.62
Conservative Opinions	.69	.52	.57
Pro-Science Beliefs	.68	.58	.65
Religious Belief	.95	.92	.95

Name	Correlation (r) with conservatism	Mean (SD)
Abortion^rev	−.46^***	54.1 (37.3)
Same Sex Marriage^rev	−.45^***	63.1 (38.1)
Military	.41^***	57.9 (31.8)
Police Authority	.36^***	52.0 (30.3)
Men in Feminism^rev	−.31^***	61.9 (29.6)
War	.30^***	55.1 (31.9)
Capital Punishment	.24^***	67.4 (31.2)
Sexism for Men	.24^***	48.7 (29.4)
Campus Free Speech^rev	−.23^***	46.3 (32.5)
Microaggressions^rev	−.15^***	58.6 (26.3)

Name	Correlation (r) with conservatism	Mean (SD)
Global warming	-.34***	74.3 (28.1)
Big Bang	-.29***	57.5 (31.8)
Evolution	-.25***	60.1 (33.1)
Old Earth	-.18***	77.5 (26.7)
Stem Cells	-.16***	71.8 (23.4)
Vaccines/Autism	-.05	65.7 (31.3)
Tech Problems	-.03	55.5 (29.7)
Modern Medicine	-.02	69.1 (26.6)
IQ Heritability	-.003	66.6 (23.9)
Detoxing	.01	31.2 (25.6)
Genetics	.02	60.4 (26.2)
GMO/Health	.03	37.9 (28.2)
Nuclear Power	.12**	48.5 (28.9)
*indicates p < .001, indicates p < .01.

On the belief that beliefs should change according to evidence: Implications for conspiratorial, moral, paranormal, political, religious, and science beliefs

Gordon Pennycook*, James Allan Cheyne# Derek J. Koehler# Jonathan A. Fugelsang#

1 Introduction

1.1 Is reasoning helpful or helpless?

1.2 Does reasoning undermine sound judgment?

1.3 Actively open-minded thinking

1.4 Current work

2 Study 1

2.1 Method

2.1.1 Participants

2.1.2 Materials

AOT-E.

Conspiracist ideation.

Paranormal belief.

God Skepticism.

Moral values.

Political ideology.

Political opinions.

Free Market Ideology.

Science beliefs.

Demographics.

2.1.3 Procedure

2.2 Results and Discussion

3 Study 2

3.1 Method

3.1.1 Participants

3.1.2 Materials

Measures identical to Study 1.

AOT-E.

Cognitive Reflection Test (CRT).

Religious belief.

Science beliefs.

Political party.

Demographics.

3.1.3 Procedure

3.2 Results and Discussion

4 Study 3

4.1 Method

4.1.1 Participants

4.1.2 Materials and Procedure

4.2 Results and Discussion

5 General Discussion

5.1 AOT-E among Democrats and Republicans

5.2 Very large effect sizes: A lesson

5.3 Limitations

6 Conclusion

References

Gordon Pennycook^*, James Allan Cheyne^# Derek J. Koehler^# Jonathan A. Fugelsang^#