Ideology, motivated reasoning, and cognitive reflection

Decision scientists have identified various plausible sources of ideological polarization over climate change, gun violence, national security, and like issues that turn on empirical evidence. This paper describes a study of three of them: the predominance of heuristic-driven information processing by members of the public; ideologically motivated reasoning; and the cognitive-style correlates of political conservativism. The study generated both observational and experimental data inconsistent with the hypothesis that political conservatism is distinctively associated with either unreflective thinking or motivated reasoning. Conservatives did no better or worse than liberals on the Cognitive Reflection Test (Frederick, 2005), an objective measure of information-processing dispositions associated with cognitive biases. In addition, the study found that ideologically motivated reasoning is not a consequence of over-reliance on heuristic or intuitive forms of reasoning generally. On the contrary, subjects who scored highest in cognitive reflection were the most likely to display ideologically motivated cognition. These findings corroborated an alternative hypothesis, which identifies ideologically motivated cognition as a form of information processing that promotes individuals’ interests in forming and maintaining beliefs that signify their loyalty to important affinity groups. The paper discusses the practical significance of these findings, including the need to develop science communication strategies that shield policy-relevant facts from the influences that turn them into divisive symbols of political identity.

1 Introduction

Ideological polarization is a conspicuous but peculiar feature of American democracy. No one is surprised when conservatives and liberals fight over tradeoffs between wealth and equality or between “law and order” and civil liberties. Differences in the value attached to such goods define those political outlooks.

Much more perplexing, however, are the ubiquity and ferocity of ideological conflicts over facts that turn on empirical evidence. Democrats (by and large) fervently believe that human activity is responsible for global warming, Republicans (by and large) that it is not (Pew Research Center, 2012). Conservatives are confident that the wastes generated by nuclear power plants can be safely disposed of by deep geologic isolation; liberals dispute that (Jenkins-Smith, Silva, Nowlin & deLozier, 2011). People who value equality and community believe that vaccinating schoolgirls against the human papilomavirus is essential to protecting women’s health—and that permitting private citizens to carry concealed hand guns increases crime. Those who value hierarchy and individualism, in contrast, reply that universal HPV vaccination will undermine young girls’ health by lulling them into unprotected sex, and that gun control increases crime by making it harder for law-abiding citizens to protect themselves (Kahan, Braman, Cohen, Gastil & Slovic, 2010; Kahan, 2010).

Political polarization on empirical issues like these occurs not only despite the lack of any logical connection between the contending beliefs and the opposing values of those who espouse them. It also persists despite apparent scientific consensus on the answers to many of these disputed questions (Lewandowsky, Gignac, & Vaughan, 2012; Kahan, Jenkins-Smith & Braman, 2011).

Decision science suggests a range of potential explanations for why dueling assertions of fact occupy frontline positions in U.S. culture wars. It is well established that members of the public rely on heuristics or mental shortcuts that can generate systematic biases in their perceptions of risk and similar facts. They also tend to seek out and assess evidence in biased patterns that reinforce the positions that they, or those who share their ideological predispositions, already hold. Some psychologists maintain, too, that these effects are intensified by dogmatism, aversion to complexity, and like traits that correlate with political conservativism and that make politically conservative individuals distinctively resistant to revising their beliefs based on empirical evidence.

All of these positions are firmly rooted in empirical study, but relatively little testing has been done on how they relate to one another. More than one plausible account exists of how the various dynamics that connect ideology and factual perceptions interact. Empirical testing of these competing surmises is necessary, not only to advance understanding of ideological polarization over policy-relevant facts but also to guide practical action aimed at mitigating it.

This paper reports the results of a study designed to contribute to the necessary testing process. Using both observational and experiment methods, the study examined alternative hypotheses relating to how three psychological dynamics—dual process reasoning, the cognitive style associated with conservativism, and motivated cognition—combine to generate polarization over risks and other policy consequential facts.

2 Empirical and theoretical background

Scholars have identified three important influences that contribute to public controversy over factual questions that turn on scientific and related forms of empirical evidence. This part briefly describes these influences and identifies alternative conjectures about how they interact with one another.

The first dynamic associated with public controversy over policy-relevant facts is heuristic-driven information processing. Long prominent in the study of psychology, dual-process theories posit two modes of information-processing: a “fast, associative” one “based on low-effort heuristics”, and a “slow, rule based” one that relies on “high-effort systematic reasoning” (Chaiken & Trope, 1999, p. ix). Many scholars attribute controversy over societal risks to the disposition of members of the public to over-rely on the heuristic-driven, “System 1” (Stanovich & West, 2000; Kahneman, 2003) reasoning style. The centrality of visceral, emotion-guided modes of perception can cause laypeople to overestimate the incidence and harm associated with more sensational risks—such as terrorist acts and gun accidents—relative to more remote, less gripping hazards such as climate change and swimming pools. Expert opinion does not reliably correct these distortions because members of the public too often lack the time or ability to engage in the more effortful, more dispassionate “System 2” style of reasoning suited to understanding the technical evidence that experts use to assess risks (Loewenstein, Weber, Hsee & Welch, 2001; Sunstein, 2003, 2006a, 2007; Weber, 2006).

The second dynamic that generates conflict over risk issues is motivated reasoning. Motivated reasoning refers to the tendency of people to conform assessments of information to some goal or end extrinsic to accuracy (Kunda, 1990; Balcetis & Dunning, 2008; Dunning, 1999; Ditto, Pizarro & Tannenbaum, 2009).

The goal of protecting one’s identity or standing in an affinity group that shares fundamental values can generate motivated cognition relating to policy-relevant facts (Cohen, 2003; Sherman & Cohen, 2006). Even among modestly partisan individuals, shared ideological or cultural commitments are likely to be intertwined with membership in communities of one sort or another that furnish those individuals with important forms of support—emotional and psychic as well as material (Green, Palmquist & Schikler, 2002). If a proposition about some policy-relevant fact comes to be commonly associated with membership in such a group, the prospect that one might form a contrary position can threaten one’s standing within it. Thus, as a form of “identity self-defense,” individuals are unconsciously motivated to resist empirical assertions—that the death penalty deters or does not deter murder, for example (Lord, Ross & Lepper, 1979), or that gun control reduces or does or does not reduce crime (Taber & Lodge, 2006)—if those assertions run contrary to the dominant belief within their groups (Cohen, Bastardi, Sherman, Hsu, McGoey, & Ross, 2007; Liu & Ditto, 2013; Munro & Ditto, 1997).

The third influence linked to such controversies is the association between ideological or cultural values and cognitive-reasoning styles. Refining the 1950s “authoritarian personality” theory of Adorno and his collaborators (1950), a substantial body of empirical study generated in the last decade has revived interest in, and compelled respectful scholarly engagement with, the hypothesis that right-wing ideology is a manifestation of settled intellectual traits such as dogmatism, aversion to complexity, and a craving for certainty or “closure” in argumentation. The cognitive style that comprises these dispositions, it is surmised, generates reflexive closed-mindedness toward empirical evidence hostile to the factual premises of policies that reflect ideologically conservative values or policy preferences (Jost, Glaser, Kruglanski & Sulloway, 2003; Jost, Nosek & Gosling, 2008; Feygina, Jost & Goldsmith, 2010).

While all of these positions are amply supported by empirical evidence, how they interact to generate polarization over empirical evidence relevant to societal risks and like facts has not been systematically explored (Jost, Hennes & Lavine, in press). Multiple alternative relationships are all plausible.

It would not be surprising, for example, to discover that the impact of heuristic-driven reasoning is the most decisive one. The predominance of System 1 reasoning among members of the general public, on this view, would account for the failure of democratic institutions to reliably converge on the best available scientific evidence on issues like climate change. Dynamics of motivated cognition would in turn help to explain the ideological character of the resulting public controversy over such evidence. Many of the emotional associations that drive System 1 risk perceptions, it is posited, originate in (or are reinforced by) the sorts of affinity groups that share cultural or ideological commitments (Leiserowitz, 2005; Sunstein, 2007). Where the group-based associations that attach to putative risk sources (guns, say, or nuclear power plants) vary, then, we can expect to see systematic differences in risk perceptions across members of ideologically or culturally uniform groups (Weber & Stern, 2011; Lilienfeld, Ammirati, & Landfield, 2009; Sunstein, 2006b; Marx, Weber, & Orlove, Leiserowitz, Krantz, Roncoli, & Phillips, 2007; Westen, Blagov, Harenski, Kilts, & Hamann, 2006). For expositional convenience, this account of how heuristic information processing, ideological predispositions, and motivated reasoning interact will be referred to as the “Bounded Rationality Position” (BRP).

An alternative position might see the reasoning-style correlates of right-wing ideology as the most consequential for disputes over decision-relevant science. Like BRP, this position would regard motivated cognition as a heuristic-driven form of reasoning. The mental dispositions that researchers have identified with conservative ideology—dogmatism, need for closure, aversion to complexity, and the like—are understood to indicate a disposition to rely predominantly on System 1 rather than more effortful System 2 forms of information processing (Stanovich, 2011, pp. 34-35). Accordingly, the impact of ideologically motivated cognition, even if not confined to conservatives, would be disproportionately associated with that ideology by virtue of the negative correlation between conservativism and the traits of open-mindedness and critical reflection that would otherwise check and counteract it (Jost, Hennes, Lavine, in press; Nam, Jost & van Bavel, 2013).

Because it predicts that motivated reasoning will be uneven across the ideological spectrum, this account can, for expositional convenience, be referred to as the “Ideological Asymmetry Position” (IAP). Relatively little experimental work specifically testing IAP exists at this point. It is mixed, with some scholars reporting results that suggest conservative subjects display greater resistance to engaging counter-attitudinal evidence than liberals do (Nam et al., 2013), but others finding that motivated reasoning can skew perceptions—including ones formed on the basis of brute sense impressions (Kahan, Hoffman, Evans, Rachlinski, & Braman, 2012)—symmetrically across groups of diverse cultural and political orientations (Crawford & Xhambazi in press; Crawford, 2012).

Finally, another position might treat identity-protective motivated reasoning as primary in significance. When individuals display identity-protective cognition, their processing of information will more reliably guide them to perceptions of fact that are congruent with their membership in ideologically or culturally defined affinity groups than to ones that reflect the best available scientific evidence (Kahan et al., 2011). Nevertheless, this form of information processing, when applied to the sorts of facts at issue in polarized policy disputes, will often make ordinary individuals better off, in terms of self-interest (Kahan, Peters, Wittlin, Slovic, Ouellette, Braman, & Mandel, 2012). Any mistake an individual makes about the science on, say, the reality or causes of climate change, the disposal of nuclear wastes, or the effectiveness of an assault-rifle ban in deterring mass shootings will not affect the level of risk for her or for any other person or thing she cares about: Whatever she, as a single individual, does and can do—as consumer, as voter, as participant in public discourse—will be too inconsequential to have an impact (Downs, 1957). But, insofar as competing positions on these issues have come to express membership in and loyalty to opposing social groups, a person’s formation of a belief out of keeping with the one that predominates in hers could mark her as untrustworthy or stupid, and thus compromise her relationships with others (Cohen, 2003). These consequences could substantially diminish her welfare—materially and psychically (Sherman & Cohen, 2002; Akerlof & Kranton, 2000; Hillman, 2010).

If we imagine that socially adaptive pressures will favor reasoning styles that maximize this form of “expressive utility” (Gigerenzer, 2000), we might, on this account, expect the use of more effortful, System 2 forms of information processing to magnify, not mitigate, ideological differences. Individuals disposed to resort to heuristic-driven, System 1 cognitive processing should not have too much difficulty fitting in: Conformity to peer influences, receptivity to elite cues, and sensitivity to intuitions calibrated by the same will ordinarily guide them reliably to stances that cohere with and express their group commitments (Zaller, 1992; Gastil, Braman, Kahan & Slovic, 2011). But if individuals are adept at using more effortful, System 2 modes of information processing, then they ought to be even better at fitting their beliefs to their group identities. Their capacity to make sense of more complex forms of evidence (including quantitative data) will supply them with a special resource that they can use to fight off counterarguments or to identify what stance to take on technical issues more remote from ones that that figure in the most familiar and accessible public discussions (Chen, Duckworth & Chaiken, 1999). More importantly still, it will make them more likely to understand the significance of competing claims, and related forms of evidence, for the status of their group, and thus be more likely to experience unconscious motivations to form identity-congruent assessments of them.

This account—which for convenience can be labeled the “Expressive Utility Position” (EUP)— inverts the relationship that BRP posits between motivated cognition and dual-process reasoning. Whereas BRP views ideological polarization as evidence of a deficit in System 2 reasoning capacities, EUP predicts that the reliable employment of more effortful, conscious information processing will magnify the polarizing effects of identity-protective cognition (Kahan, Peters et al. 2012).

EUP also is inconsistent with IAP. Both BRP and IAP assume motivated reasoning reflects a deficit in the capacity or disposition to engage in effortful information processing of the sort characteristic of System 2. EUP (like BRP) is agonistic on whether reliance on heuristic, System 1 reasoning dispositions will be correlated with ideological or other values. But because EUP asserts that the more effortful, conscious forms of reasoning characteristic of System 2 magnify identity-protective cognition, EUP does not see the correlations featured in IAP as implying that motivated reasoning should be disproportionately concentrated in conservatives.

3 Study: sample, design, and hypotheses

A study was designed to investigate how heuristic-driven information processing, the reasoning-style correlates of conservativism, and motivated cognition interact. The study included both an observational component, which measured the cognitive reasoning dispositions of subjects of diverse ideologies; and an experimental one, which assessed the interaction between cognitive-reasoning dispositions, subjects’ ideologies, and their display of ideologically motivated reasoning. The design of both components was crafted to pit hypotheses distinctive of BRP, IAP, and EUP against one another.

3.1 Sample

Subjects for the study consisted of a nationally diverse sample of 1750 U.S. adults. The subjects were recruited by YouGov, a firm that uses on-line sampling methods suited for academic, political, and commercial public opinion research. The sample was stratified to reflect demographic characteristics, including political outlooks and affiliations, representative of the United States general population. The sample was 54% female, and the average age of the subjects was 52 years. Seventy-six percent of the subjects were white, and 11% African-American. Twenty-eight percent of the sample self-identified as Republican, 36% as Democrat, and 30% as independent. Twenty-five percent identified themselves as either “Liberal” or “very Liberal”: 37% as “Conservative” or “very Conservative”; and 29% as “Moderate.” The mean education level was “some college”; the mean annual income was $40,000 to $49,000. The study was administered in July 2012.

3.2 Observational component

a. Measures.

Subjects furnished standard demographic data. Party self-identification (“dem_repub”) was measured with a seven-point item (“Strong Democrat, Democrat, Independent Lean Democrat, Independent, Independent Lean Republican, Republican, Strong Republican”). Political ideology (“libcon”) was measured with a five-point item (“Very liberal”; “Liberal”; “Moderate”: “Conservative”; “Very Conservative”).

Responses to these two items (standardized and summed (Smith, 2000)) formed a reliable aggregate scale (α = 0.82). The scale was constructed so that its center was equivalent to a “neutral” political orientation. This was accomplished by centering the standardized scale components at their ordinal midpoints (“4” for dem_repub, and “3” for libcon) rather than their means and centering the composite scale formed by their sum at its midpoint as opposed to its mean. Labeled “Conserv_Repub,” the scale was centered at “0” to facilitate ease of interpretation, and oriented so that negative values denoted a disposition toward Democratic Party affiliation, and liberal ideology and positive values a disposition toward Republican Party affiliation and conservative ideology.

Subjects also completed the Cognitive Reflection Test (CRT). The CRT is a three-question test that is designed to measure the disposition to engage in the conscious and effortful form of information processing associated with System 2 as opposed to the heuristic-driven form associated with System 1 (Frederick, 2005). A performance-based measure, the CRT has been shown to be a strong predictor of cognitive biases associated with over-reliance on heuristics (Hoppe & Kusterer, 2011)—indeed, a stronger one than numerous self-report measures of critical reasoning (Toplak, West & Stanovich, 2011; Liberali, Reyna, Furlan & Pardo, 2011).

The CRT is a demanding test, and it is not unusual for a high proportion of a general population sample to answer none of the questions correctly (Weller, Dieckmann, Tusler, Mertz, Burns & Peters, 2012; Campitelli & Labollita, 2010). The mean score for subjects in this study was 0.65 (SD = 0.95).

b. Hypotheses.

As will be explained presently, results from the observational component of the study furnish maximum insight in conjunction with those of the experimental component. However, IAP suggests one fairly straightforward hypothesis relating to the observational study considered alone: that a “right-wing” outlook should be associated with a lower CRT score.

IAP is based on survey correlations between self-identifying as “Republican” or “conservative” and traits such as dogmatism, fear of complexity, and need for closure (Jost et al., 2003). Because these reasoning traits are opposed to reflection and related forms of critical thinking, one might expect a right-wing disposition to be negatively correlated with performance on the CRT. Indeed, researchers who study cognition routinely treat CRT scores as a performance-based alternative to or “behavioral” corroborator of the same self-report measures featured in research that identifies conservative ideology as distinctively hostile to reflective engagement with counter-attitudinal evidence (Stanovich, West & Toplak, 2011, p. 348 n.5; Pennycook Cheyne, Barr, & Koehler, 2013; Boschetti, Richer, Walker, Price, & Dutra, 2012; Iyer, Koleva, Graham, Ditto, & Haidt, 2012; Pennycook, Seli, Koehler, & Fugelsang, 2012).

Researchers investigating parallel claims about religiosity—that it is either a consequence or a cause of aversion to complexity, fear of uncertainty, and resistance to reflectively engaging with counter-attitudinal evidence—have reported finding that religiosity is negatively correlated with CRT (Pennycook et al., 2013; Shenhav, Rand & Greene, 2011; Pennycook Cheyne, Seli, Koehler & Gugelsang, 2012; Gervais & Norenzayan, 2012). A finding that conservative ideology is negatively associated with CRT as well would thus help corroborate the literature on which IAP is based.

Only a modest amount of work exists on the relationship between CRT and political ideology, but it is consistent with the IAP hypothesis. Two studies report finding CRT scores to be slightly lower in self-identified conservatives than in self-identified liberals (Iyer et al., 2012; Pennycook et al., 2012).

Both of these studies, however, were based on non-representative samples. The subjects in the study reported in Iyer et al. (2012) were recruited from visitors to a web site that features research on the relationship between political outlooks and cognitive styles. As an incentive to participate, subjects were told they would be supplied with a permanently maintained moral personality “profile” that would enable them to compare their scores on various measures to those of other participants, past and future. One might expect subjects visiting such a site and responding to such an offer to overrepresent highly reflective individuals. Indeed, study subjects of all ideologies achieved remarkably high mean CRT scores (Iyer et al., 2012, Table 3)—ones that not only exceeded those observed in general population samples but that matched or exceeded those recorded among students at elite universities such as Carnegie Mellon, Princeton, and Harvard (Frederick, 2005, Table 1). Thus, one could reasonably question whether data from a sample this distinctive supports inferences about the relationship between cognitive style and political outlooks in the public at large.

The sample in Pennycook et al. (2012) consisted of members of Amzaon.com’s “Mechanical Turk” workforce. Such samples, which are assembled without any recruitment or stratification procedures to assure representativeness, have been reported to be highly skewed toward liberal respondents (Richey & Taylor, 2012). The sample in Pennycook et al. (2012, p. 5) reflected such a skew.¹ It is at least possible that a sample drawn from this source, members of whom engage in myriad highly routinized tasks for an average wage of $1.40 per hour (Jaquet, 2011), might underrepresent reflective conservatives. Researchers have also reported that many Mechanical Turk study subjects have been exposed to objective performance measures like those featured in CRT on multiple occasions (Chandler, Mueller, & Paolacci, 2013), an experience that might render their scores unreliable. Finally, there is reason to believe that Mechanical Turk workers routinely misrepresent their nationality: Researchers report that “substantial” numbers of Mechanical Turk workers recruited for studies open only to U.S. “workers” nevertheless participate in them through foreign internet service providers (Shapiro, Chandler & Mueller, 2013). For these reasons, one might also reasonably question whether studies based on Mechanical Turk workers support reliable inferences about cognitive style and ideology in the U.S. general population.

In sum, the negative correlations between CRT and conservativism reported in Iyer et al. (2012) and Pennycook et al. (2012) are definitely plausible and very much consistent with the reported correlations between conservativism and cognitive style in the scholarship that supports IAP. But it remains useful to explore the relationship between CRT and ideology in samples recruited and stratified to assure national representativeness. The absence of such a finding would be surprising and would complicate interpretation of the finding that conservativism is negatively associated with various subjective measures of heuristic information processing.

A finding that CRT scores correlate with conservative ideology in the general population would have inconclusive significance for BRP and EUP. BRP does not suggests any basis to expect an ideological asymmetry in the dynamics that result in political conflict over policy-consequential facts, but does not necessarily rule it out. EUP, in contrast, does assert that such dynamics should be symmetric. However, it does not identify the source of ideological conflict over fact with the predominance of heuristic-driven, System 1 forms of information processing.

3.3 Experimental component

a. Stimuli and measures.

In the experimental component of the study, subjects reported their own perceptions of the validity of the CRT upon completion of it. They did so by indicating (on a six-point item) their level of agreement or disagreement with the statement “I think the word-problem test I just took [i.e., the CRT test] supplies good evidence of how reflective and open-minded someone is” (CRT_valid).²

Subjects responded to this item after being assigned to one of three experimental conditions. In the “control” condition, subjects were advised simply that “psychologists believe the questions you have just answered measure how reflective and open-minded someone is.” In the “skeptic-is-biased” condition, subjects were told in addition that “in one recent study, a researcher found that people who accept evidence of climate change tend to get more answers correct than those who reject evidence of climate change,” a “finding [that] would imply that those who believe climate change is happening are more open-minded than those who are skeptical that climate change is happening.” In contrast, in the “believer-is-biased” condition, subjects were advised that “in one recent study, a researcher found that people who reject evidence of climate change tend to get more answers correct than those who accept evidence of climate change,” a “finding [that] would imply that those who are skeptical climate change is happening are more open-minded than those who believe that climate change is happening.”

b. Hypotheses.

Because open-mindedness is a positive characteristic, individuals presumably have an emotional stake in believing that people who subscribe to their own ideology are more open-minded and reflective, or at least not less, than those who subscribe to an opposing one. Here subjects were presented evidence relevant to that issue: the respective performance on an “open-mindedness test” of people who either accepted or rejected a position strongly associated with membership in the subjects’ own ideological groups. The subjects were in fact supplied relatively spare information about the validity of CRT: only the representation that psychologists view it as valid, plus the subjects’ own experience in having just taken it. The prospect that they would engage in motivated reasoning, though, supplied a basis for believing they would treat that evidence as establishing the test’s validity conditional on whether doing so would gratify or disappoint their stake in believing that members of their ideological group were more open-minded than members of an opposing one (Cohen, 2003).

At a very concrete level, this design can be viewed as a model of how ordinary people process information about studies of the ideological correlates of cognitive reasoning styles (e.g., Mooney 2012). Such readers will have little to go on besides scholars’ or commentators’ representations that the tests of open-mindedness featured in such studies are valid. If such readers are inclined to credit such representations only when the studies’ results gratify their interest in forming and maintaining the belief that people who share their own ideology are more open-minded, then their assessments of that research will itself be biased by ideologically motivated reasoning.

But, abstracting from the particulars, the study design can be also thought of as modeling how ideologically motivated reasoning might bias considerations of empirical evidence generally. On policy debates over matters as diverse as climate change, gun control, the death penalty, and fiscal policy, ordinary citizens are presented with evidence, often in the form of second-hand characterizations of the findings of “scientific studies”. If their assessments of the validity of such evidence is conditional on its fit with their ideological predispositions, then citizens will not converge on the best available evidence but rather will polarize on policy-relevant facts (Lord, Ross & Lepper 1979; Kahan, Jenkins-Smith & Braman, 2011).

BRP, IAP, and EUP all predict motivated reasoning in this study. They generate different hypotheses, however, about the form that such reasoning will take.

IAP predicts that the observed motivated reasoning should be stronger in conservative subjects. Reflecting their disposition toward dogmatism and closed-mindedness, relatively right-wing subjects should be more inclined to see CRT as a valid test in the believer-is-biased condition than in the skeptic-is-biased condition. Scholars who have presented evidence of the cognitive-style correlates of ideology depict more liberal or left-wing individuals as more open-minded and reflective and thus less subject to motivated cognition. Accordingly, in the study, this work would predict that relatively left-wing subjects’ assessments of the validity of CRT should be comparable in both the skeptics-biased and believer-is-biased condition. This finding would be a particularly compelling affirmation of IAP, moreover, in conjunction with a finding in the observational component of the study that right-wing dispositions correlate with a lower CRT score.

BRP understands motivated reasoning to be an artifact of the disposition to use low-level, System 1 information processing. Unlike IAP, BRP does not (or does not necessarily) predict that motivated reasoning will be ideologically asymmetric. But BRP does predict that it will be higher among subjects who score relatively low in CRT than it will be in those who score relatively high. On this account, any ideological polarization observed in the skeptic-is-biased and believer-is-biased conditions should dissipate as CRT increases.

EUP supports predictions strikingly different from those of either IAP or BRP. Not only does EUP fail to predict ideological asymmetry in motivated reasoning. It predicts that the tendency of both right-wing and left-wing subjects to form ideologically congruent assessments of the “validity” of CRT will increase with their CRT scores. All subjects, EUP posits, will experience psychological pressure to fit their perceptions of the CRT’s validity to their interest in believing that members of their group are more open-minded about evidence on climate change. Nevertheless, their success in achieving this end will depend on their comprehension of the questions being posed and their appreciation of what differing answers signify about the open-mindedness of individuals who share their ideologies. Because subjects who are high in CRT assess information more methodically and reflectively, they are less likely to misunderstand the question, and thus less likely to avoid the unconscious pressure to fit their assessments of the evidence at hand to the conclusion that fits their expressive interests.

3.4 Statistical power and missing data

Certain of the competing hypotheses associated with BRP, IAP, and EUP turned on whether or not an observational correlation or experimental effect would be observed. The strength of inferences drawn from “null” findings depends heavily on statistical power. The large size of the sample furnished adequate power to detect even small effect sizes (e.g., r = 0.10) with a probability well over 0.80 at p ≤ .05 (Cohen, 1988). As a result, findings of nonsignificance could be equated with lack of effect with low risk of Type II error (Streiner, 2003). To assure full exploitation of the power associated with the large sample size, missing data were replaced by multiple imputation (King, Honaker, Joseph & Scheve, 2001; Rubin, 2004).³

4 Results

4.1 Observational component

Subjects’ CRT scores were regressed (Table 2) against political orientations and other individual characteristics known to be associated with CRT performance, including gender, race, education, income, and religiosity (which was measured with a composite scale that aggregated self-reported church attendance, frequency of prayer, and perceived importance of God, α = 0.82). The coefficient for Conserv_Repub was positive—indicating that CRT scores increase as subjects become progressively more right-wing in their orientation—but nonsignificant both when treated as a zero-order predictor (b = 0.07, p = 0.15) and when included in a model that controlled for other characteristics (b = 0.05, p = 0.33). The other characteristics, in contrast, all showed significant, independent effects on CRT scores (Table 2).

The Conserv_Repub scale is a more reliable measure of the subjects’ ideological dispositions than either of its components alone, and thus supports more reliable inferences about the relationship between political outlooks and CRT scores (Gliem & Gliem, 2003). Nevertheless, to promote comparability of the results reported in this study and ones that use only a single political orientation measure, the relationships between subjects’ scores and aspects of each component of Conserv_Repub were also examined.

The correlation with self-reported liberal-conservative ideology (libcon) was negative—indicating a decline in score as conservativism increases. The size of the effect, however, was only trivially different from zero and statistically nonsignificant (r_s= -0.02, p = 0.45). When the subjects who self-identified as “Moderates” were excluded, and the remainder split into groups who identified as either “Very liberal” or “Liberal,” on the one hand, or “Very conservative” or “Conservative,” on the other, there was a slightly larger but still statistically nonsignificant difference (Δ M = 0.09, t = 1.41, p = 0.16) in the mean scores of “liberals” (M = 0.75, SEM = 0.05) and “conservatives” (M = 0.67, SEM = 0.04).

The correlation with self-reported party affiliation (r_s= 0.08, p < 0.01) was positive and significant, indicating that CRT scores increased with the strength of subjects’ identification with the Republican party on dem_repub, the 7-point measure of partisan identification. When subjects who self-identified as either “Independents” or “Independents” who “lean” either Democrat or Republican were removed, and the remainder split into ones who identified as either “Democrat” or “Republican,” the difference in the mean score of “Republicans” (M = 0.66, SEM = 0.04) and “Democrats” (M = 0.52, SEM = 0.03) was also statistically significant (Δ M = 0.14, t = 2.48, p < 0.05). The gap between “Republicans” (M = 0.71, SEM = 0.04) and “Democrats” (M = 0.59, SEM = 0.03) remained statistically significant (Δ M = 0.13, t = 2.54, p < 0.05) when the scores of subjects who identify as “Independents” but “lean” toward one or the other of the major parties were treated as partisans (Petrocik, 2009).

Based on work relating personality-trait and self-reported measures of cognitive style to ideology, IAP predicted that CRT scores would be negatively correlated with right-wing ideology. This hypothesis was not confirmed.

4.2 Experimental component

a. Ideologically motivated reasoning generally.

Consistent with BRP, IAP, and EUP, subjects displayed ideologically motivated reasoning in their assessments of the validity of the CRT (Figure 1, Panel (A)). In the control condition, right-wing and left-wing subjects (identified by their scores relative to the mean on Conserv_Repub), formed comparable judgments of the validity of the CRT as a measure of how “reflective and open-minded” people are (M = 3.0, SEM = 0.2, for both; Δ M = 0.08, t = 0.59, p = 0.56). Those assigned to the “skeptic-is-biased” condition, in contrast, divided on ideological lines: right-wing subjects were less inclined to treat the CRT as valid (M = 2.4, SEM = 0.2), and left-wing ones more inclined to do so (M = 3.5, SEM = 0.2), when told that climate-change skeptics generally score lower on the test. This effect was reversed in the “believer-is-biased” condition: told that climate skeptics generally score higher on the test, relatively right-wing subjects were now more inclined (M = 3.1, SEM = 0.2), and relatively left-wing ones less, to judge the CRT to be valid (M = 2.6, SEM = 0.2).

b. Symmetry.

Figure 1 plots subject responses using a locally weighted regression smoother. This technique, which traces the profile of the data in detail without making any particular assumptions about their distribution, is an appropriate device both for graphical summary assessments of the raw data and for determining the appropriate form (linear or curvilinear) of the statistical model to apply in analyzing them (Jacoby, 2000).

Visual inspection of Figure 1 does not suggest that the observed motivated-reasoning effect increased as subjects became more right-wing. Indeed, liberal Democratic subjects were inclined to discount the validity of the CRT in the “believer-is-biased” condition relative to the “skeptic-is-biased” condition (Δ M = 0.86, t = 6.37, p < 0.01) by an amount slightly but nonsignificantly larger (Δ M = 0.11, t = 0.57, p = 0.57) than the amount by which conservative Republican ones were inclined to discount it in “skeptic-is-biased” condition relative to the “believer-is-biased” one (Δ M = 0.75, t = 5.39, p < 0.01).⁴

c. Interaction with CRT.

The final experimental hypotheses were BRP’s prediction that ideological polarization should abate as individuals become more disposed to use System 2 processing and EUP’s prediction that motivated reasoning would instead increase. As is evident in Panels (B)−(D) of Figure 1, the tendency of subjects (varying in political dispositions) to form ideologically congenial assessments of the validity of the Cognitive Reflection Test became progressively more pronounced as subjects’ CRT scores increased. The effect appears most dramatic in the “believer-is-biased” condition.

To confirm and test the statistical significance of this effect, a regression model reflecting the interaction of CRT and subjects’ ideological dispositions was fit to the data. The results are reported in Table 3, Model 2. This model fit the data significantly better than one lacking terms reflecting the hypothesized interaction (Δ LR χ 2 = 12.64(6), p = 0.01). The coefficient for the interaction term added to assess the relationship between CRT scores and assignment to the “believer-is-biased” condition (Con_Rep_x_zCRT_x_believer), was positive and significant (b= 0.30, p < 0.01). Thus, in the “believer-is-biased” condition, subjects of opposing ideologies polarized to a greater extent as their CRT scores increased. The coefficient for the interaction term added to assess the relationship between CRT scores and assignment to the “skeptic-is-biased” condition (Con_Rep_x_zCRT_x_skeptic) was only trivially different from zero and nonsignificant (b = −0.03, p = 0.76). Thus, in the “skeptic-is-biased” condition, the degree of ideological polarization did not meaningfully vary in relation to subjects’ CRT scores.

Graphic illustration enables practical assessment of this interaction effect. Figure 2 plots the predicted probability that subjects of opposing political outlooks with either “low” or “high” in CRT scores would agree (either slightly, moderately, or strongly) that the CRT is a valid measure of open-mindedness in each of the experimental conditions. The estimates were formed using regression Model 2 in Table 3. For “Liberal Democrat” and “Conservative Republican,” the value assigned to Conserv_Repub was set at the level a subject would have scored had she selected either “liberal” and “Democrat” or “conservative” and “Republican” on the 5-point liberal-conservative ideology and 7-point partisan self-identification items, respectively. For “low CRT,” the value assigned to the CRT predictor was 0 correct, a score that would place those individuals in the bottom 60% within the sample. For “high CRT,” the predictor was set at 2 correct, a score matched or exceeded by 20% of the sample.

As can be seen, the impact of CRT performance on motivated reasoning is concentrated in the “believer-is-biased” condition. The difference in predicted probability of judging the CRT to be “valid” in that condition is 28 percentage points (± 19%, CL = 0.95) greater for high-CRT partisans than it is for low-CRT partisans, whose likelihoods of judging the CRT to be “valid” do not meaningfully differ in that condition. Both high-scoring and low-scoring partisans are highly likely to disagree about the validity of the CRT in the “skeptic-is-biased” condition. The likelihood that high-scoring ones will disagree is higher than is the likelihood that low-scoring ones in that condition will, but the difference is modest and not significant at 0.95 level of confidence (8%, ± 11%).

These results are more consistent with EUP than BRP. Whereas the former predicted that motivated reasoning would abate as CRT increases, the latter predicted that it would not but would instead intensify. Such intensification did occur, but only in one of the experimental conditions.

5 Discussion

5.1 Summary of results

The motivation (consciously, at least) for this study was to assess three dynamics understood to be relevant to political contestation over facts that turn on scientific or related forms of empirical evidence: heuristic-driven reasoning, the cognitive-style correlates of political conservativism, and motivated cognition. For that purpose, the study measured the cognitive reflection of individuals of diverse ideologies and also tested how willing they were to entertain evidence that those who disagreed with their ideological group’s position on a charged issue—climate change—were themselves open-minded and reflective.

The study considered three conjectures about how heuristic-driven reasoning, ideological predispositions, and motivated reasoning relate to one another. The first—the “Bounded Rationality Position” or BRP—asserts that public conflict over risk and other policy relevant facts is a consequence of the predominance of heuristic-driven, System 1 information processing, which interferes with the public’s understanding of complicated evidence and motivates it to assess evidence consistently with cultural or ideological predispositions (e.g., Marx et al. 2007; Weber & Stern, 2011).

The study did not support BRP. On the contrary, the experimental component of the study demonstrated that the disposition to engage in conscious and effortful System 2 information processing—as measured by the Cognitive Reflection Test (CRT)—actually magnifies the impact of motivated reasoning.

The study results were also inconsistent with the “Ideological Asymmetry Position” or IAP. IAP attributes public conflict over policy-relevant facts to an affinity between conservative ideology and traits such as dogmatism and aversion to complexity, which fuel the motivated rejection of evidence incompatible with conservative policy aims (Jost et al., 2003).

The study findings pose two distinct challenges to IAP. First, the study found no meaningful correlation between right-wing ideology and CRT performance, an objective measure of reflection that has been shown to be one of the strongest predictors of conscious, effortful information processing. The CRT is routinely treated as a “stronger” performance-based alternative to (Pennycook et al. 2013) or “behavioral” corroborator of (Iyer et al., 2012) Need for Cognition, Need for Closure, dogmatism, fear of complexity and like measures (Stanovich, West & Toplak, 2011; Boschetti et. al., 2012), which are the basis of the research that finds conservatives to be distinctively resistant to counter-attitudinal evidence. The CRT has been used to test and generate evidence supportive of the parallel claim that religiosity is associated with resistance to scientific and other forms of empirical evidence. It is thus puzzling that CRT did not correlate negatively with conservativism (and indeed correlated positively with self-identification with the Republican Party) in the large nationally representative sample used in this study. This result is at odds with those based on nonrepresentative samples.

Second, this study used an experimental design and a statistical-testing strategy specifically fitted to evaluating the assumption that the force of ideological motivated reasoning varies in intensity across the right-left political spectrum. It found that, when assessing evidence of the other group’s propensity to consider evidence in an open-minded and reflective way, liberals and conservatives were uniformly prone to ideologically motivated reasoning. This result reinforces those observed in other studies that have reported finding that motivated reasoning is symmetric across groups of opposing political and cultural outlooks.⁵

The study also examined one other account of political polarization over risk and related policy-relevant facts: the “Expressive Utility Position” or EUP. The study results were largely in accord with this view.

EUP alone predicted both that ideologically motivated reasoning would be symmetric and that it would be amplified by higher CRT scores. Those hypotheses reflect a theory that sees ideologically motivated cognition not as a reasoning deficiency but as a reasoning adaptation suited to promoting the interest that individuals have in conveying their membership in and loyalty to affinity groups central to their personal wellbeing. Because individuals make use of this form of information-processing to protect their stake in all manner of groups—including ones not connected to politics (Sherman & Cohen, 2006; Hastorf & Cantril, 1954)—there is no reason to expect it to be more pronounced among people who subscribe to any particular ideology. In addition, because this dynamic reliably guides individuals to assign information the significance that advances their personal ends (albeit not their collective ones (Kahan, 2012b)), there is good reason to expect it to be used with even greater precision and accuracy by individuals disposed to engage in conscious, effortful forms of information processing.

The only discrepancy between the study results and EUP was the limited scope of the interaction between cognitive reflection and motivated reasoning. Higher CRT scores meaningfully amplified motivated reasoning in only one of the two experimental conditions that presented evidence that was expected to polarize (and did polarize) subjects of opposing political outlooks.

While not hypothesized, this result is not incompatible with EUP. The charge that “climate-change skeptics” are “closed minded” and “biased” is likely more familiar to people than the charge that “climate-change believers” are. The wording of the information furnished to subjects was in fact fairly dense. If subjects low in CRT had difficulty with it or were reluctant to expend the mental effort to parse it, they would have been more likely to misunderstand it when it conveyed the less familiar claim (“believers are closed minded”) than when it conveyed the more familiar one (“skeptics are closed minded”). Subjects with higher CRT scores were more likely to grasp the surprising meaning conveyed by the instruction in the “believer-is-biased” condition, making them more prone than low-scoring ones in that condition to form an appraisal of the test’s validity that conformed to their ideological predispositions.

5.2 Implications for ongoing study of dual-process and motivated reasoning

The status of motivated reasoning within dual process reasoning theories has not been studied extensively by social psychologists. Many commentators have assumed—not implausibly—that ideologically motivated cognition is a manifestation of unconscious, heuristic-driven reasoning processes amenable to being overridden by dispositions that promote reflection and critical engagement with counter-attitudinal evidence (e.g., Lilienfeld et al. 2009; Sunstein, 20006b; Westen et al., 2006). The results of this study cast doubt on this claim.

It should be stressed, however, that this result is not in tension with dual-process reasoning theories generally, but only with a particular surmise about how a particular cognitive dynamic should be assimilated to such theories. Indeed, in work that actually predates the adaptation of the now-familiar “System 1/System 2” framework, at least some dual-process theorists took the position that motivated reasoning should be expected to characterize higher-level, systematic forms of cognition as well, particularly when the motivated use of conscious and effortful processing could be expected to advance an agent’s ends in maintaining his or her connection to an identity-defining group (Cohen, 2003; Chen & Chaiken, 1999; Chaiken & Maheswaran, 1994; Giner-Sorrolla & Chaiken, 1997). The current study, by generating evidence that the disposition to engage in effortful information processing can amplify motivated reasoning, is more consistent with the latter position.

Indeed, these findings informed the “expressive utility position” tested in the study. That same hypothesis, moreover, figured in a previous observational study, which found that cultural polarization over climate-change and nuclear-power risks are greatest among individuals who are higher in scientific literacy and in numeracy, a technical-reasoning measure of which CRT is a subcomponent (Kahan, Peters et al. 2012). The experimental results of the present study help corroborate the inference that the correlations reported in that earlier study are property attributed to the power of motivated cognition to penetrate the forms of information processing associated with Kahneman’s “System 2” form of reasoning.

Of course, the findings of this single study do not conclusively demonstrate that the disposition to engage in reflective rather than heuristic-information processing invariably magnifies ideologically motivated reasoning. Empirical studies, when valid, merely supply more evidence to believe or disbelieve a hypothesis, the truth of which must be assessed on the basis of all the valid evidence at hand and thereafter reassessed on the basis of any evidence generated by future valid studies (Popper, 1959).

Specific to this particular study, there are many reasons not to consider the issue of ideologically motivated reasoning and reflective information-processing to be “closed”. The most important of these is continuing debate over the adequacy of existing measures of reflective information-processing and critical-thinking dispositions generally. One issue concerns the relative validity of self-reporting measures (such as “Need for Cognition” and “Need for Closure”), on the one hand, and objective measures such as the CRT, on the other.

Another concerns the strength of CRT relative to alternative objective measures. Because CRT is such a demanding test—because it is not uncommon, in a general population sample, for over a third of the sample to get zero answers correct, and for less than 10% to answer all three questions correctly—CRT is of limited value in explaining variance among a large portion of the population. Numeracy is superior to CRT in this respect, and is a stronger predictor of vulnerability to cognitive biases, likely for this reason (Liberali et al., 2011; Weller et al., 2012). But as indicated, numeracy too has been found to predict greater, not diminished, political polarization over risks like the ones associated with climate change and nuclear power (Kahan, Peters et al., 2012).

Exactly what both CRT and numeracy measure, however, is also contested (Liberali et al., 2011). In particular, whether either measures critical reflection directly or instead only indirectly—as a correlate of a more general proficiency in mathematical computation or even a general form of intellectual capacity—continues to be debated. Some researchers have concluded that CRT is diagnostic of all manner of critical thinking (Toplak, West & Stanovich, 2011), including the motivation to re-examine existing beliefs (Campitelli & Labollita, 2010; Stanovich, West & Toplak, 2011) associated with the Actively Open-minded Thinking construct (AOT; Baron, 2008) or reflection/impulsivity (Baron, Badgio & Gaskins, 1986). But whether CRT indeed measures willingness to revise strongly held beliefs, or does so as well as a measure specifically geared toward AOT, is not resolved. Necessarily unresolved, too, then, is whether a study of motivated reasoning that investigated its interaction with a suitable AOT scale (e.g., Haran, Ritov & Mellers, 2013) would find that the form of critical thinking that scale measures likewise magnifies ideologically motivated reasoning in the manner that CRT, numeracy, science literacy (Kahan, Peters et al., 2012), and political knowledge (Arceneaux & Johnson, 2013; Taber & Lodge, 2006) all do or instead diminishes it (Baron, 1995).

It thus makes sense to continue intensive study of the relationship between ideological polarization and information processing using a variety of measures, a strategy that recognizes the role that convergent validity always plays in insuring against the necessary imperfection of any single measure or method in the social sciences. Indeed, if a variety of measures and methods do converge on the conclusion that ideologically motivated reasoning is impervious to the habits of mind most strongly associated with valid, evidence-based inference, such a program of study is the one most likely to broaden insight into how the conditions that generate an influence this inimical to reason might be counteracted (Fernbach, Rogers, Fox & Sloman, 2013; Lavine, Johnston & Steenbergen, 2012).

5.3 Implications for counteracting ideologically motivated reasoning

The goal of empirically investigating the sources of political conflict over risk and other policy-consequential facts is not merely to explain this phenomenon but also to aid in discovery of devices for mitigating it. The study described in this paper makes a contribution to that end as well.

It does this primarily by helping to inform hypotheses about how such dynamics might be combated. Many scholars have suggested “debiasing” strategies aimed at correcting the distorting effect of System 1 reasoning on public perceptions of risk (Lilienfeld et al., 2009; Jolls & Sunstein, 2006). Because such distortions are real—and substantially interfere with human wellbeing in myriad domains—pursuit of System 1 debiasing techniques is unquestionably important. Nevertheless, if, as the present study implies, ideologically motivated cognition is not a consequence of an over-reliance on heuristic reasoning, then System 1 debiasing strategies should not be expected to abate polarization over climate change, nuclear power, gun control, the HPV vaccine or like issues (Kahan, Slovic, Braman & Gastil, 2006).

What is needed instead are interventions that remove the expressive incentives individuals face to form perceptions of risk and related facts on grounds unconnected to the truth of such beliefs (Lessig, 1995). Extending the analysis of previous papers, this one has suggested that ideologically motivated reasoning is in fact expressively rational at the individual level, because it conveys individuals’ membership in and loyalty to groups on whom they depend for various forms of support, emotional, material, and otherwise (Hillman, 2010; Akerlof & Kranton, 2000).

This account, however, presupposes that beliefs on risks and related facts bear social meanings—that they are, in fact, generally understood (tacitly, at least) to convey that the individuals who espouse them are committed to one group rather than another (Cohen, 2003). Not all risks and policy-relevant facts have this quality; indeed, relatively few do, and on the vast run of ones that do not (e.g., that pasteurization removes infectious agents from milk; that fluoridation of water fights tooth decay; that privatization of the air-traffic control system is inimical to air safety), we do not observe significant degrees of ideological or cultural polarization.

There is little reason to believe, moreover, that the meanings of highly contested facts are insusceptible of revision in a manner that would disconnect particular positions on them from membership in identity-defining groups. One can understand the historical shift in public opinion toward the risks posed by cigarettes (including third-party ones from passive smoke exposure or from the societal expenditures necessary to treat individuals with lung cancer) as having been mediated by informational campaigns aimed at altering the positive meanings that dismissing evidence of the health hazards of smoking expressed in certain subcommunities (United States Public Health Service, 2000; Gusfield 1993).

The expressive account of ideological polarization, then, underscores the value of forming and testing hypotheses about how to regulate the social meaning of risks and related policy-relevant facts. Indeed, research focusing on forecasting techniques for identifying technologies vulnerable to polarizing meanings, on governmental processes for protecting the “science communication environment” from influences that cause such meanings to take hold, and on framing and other strategies for cleaning up that environment once it has been contaminated with polarizing meanings (Kahan, 2012b), is already well underway (Corner, Whitmarsh & Xenias, 2012; Druckman & Bolsen, 2012, 2011, Nisbet & Scheufele, 2009).

5.4 Conclusion: The tragedy of the science communications commons

What might be characterized as the “expressive rationality” of ideologically motivated reasoning (Kahan, Peters et al., 2012) is intrinsic to a collective action problem (Lessig, 1995). When societal risks become suffused with antagonistic social meanings, it is (often if not always, and with respect to many if not all issues) individually rational for ordinary members of the public to attend to information in a manner that reliably connects them to the positions that predominate in their identity-defining groups. Nevertheless, if ideologically diverse individuals all follow this strategy simultaneously, they will be collectively worse off, since under these conditions, democratic institutions are less likely to converge, or to converge as rapidly as they otherwise would, on policies that reflect the best available evidence on how to protect everyone from harm. But because what any ordinary individual believes about policy will not make a difference, the collective irrationality of ideologically motivated reasoning does not by itself create any reliable pressure or mechanism to induce individuals to process information in a different, and morally and politically superior, way.

Overcoming ideological polarization over societal risks and related facts thus demands collective action specifically geared at dissolving this “tragedy of the science communications commons” (following Hardin, 1968). Supplying the knowledge needed to guide such action is a contribution that decision science is uniquely poised to make.

References

Adorno, T. W. (1950). The Authoritarian personality (1st ed.). New York: Harper.

Akerlof, G. A., & Kranton, R. E. (2000). Economics and identity. The Quarterly Journal of Economics, 115, 715–753.

Arceneaux, K., & Johnson, M. (2013). Changing minds or changing channels? Partisan news in the age of choice. Chicago: University of Chicago Press.

Balcetis, E., & Dunning, D. (2006). See what you want to see: Motivational influences on visual perception. Journal of Personality and Social Psychology, 91, 612–625.

Baron, J. (1995) Myside bias in thinking about abortion. Thinking and Reasoning, 1, 221–235.

Baron, J. (2008). Thinking and deciding (4th ed.). New York: Cambridge University Press.

Baron, J., Badgio, P., & Gaskins, I. W. (1986). Cognitive style and its improvement: A normative approach. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence, Vol. 3, pp. 173–220. Hillsdale, NJ: Erlbaum.

Boschetti, F., Richert, C., Walker, I., Price, J., & Dutra, L. (2012). Assessing attitudes and cognitive styles of stakeholders in environmental projects involving computer modelling. Ecological Modelling, 247, 98–111.

Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with judgments and choices. Judgment and Decision Making, 5, 182–191.

Chaiken, S., & Maheswaran, D. (1994). Heuristic processing can bias systematic processing - effects of source credibility, argument ambiguity, and task importance on attitude judgment. Journal of Personality and Social Psychology, 66, 460–473.

Chaiken, S., & Trope, Y. (1999). Dual-process theories in social psychology. New York: Guilford Press.

Chandler, J., Mueller, P., & Paolacci, G. Methodological concerns and advanced uses of crowdsourcing in psychological research (2013). Behavior Research Methods. Advance on-line publication. http://link.springer.com/article/10.3758/s13428-013-0365-7.

Chen, S., Duckworth, K., & Chaiken, S. (1999). Motivated heuristic and systematic processing. Psychological Inquiry, 10, 44–49.

Cohen, G. L. (2003). Party over policy: The dominating impact of group influence on political beliefs. Journal of Personality and Social Psychology., 85, 808–822.

Cohen, G. L., Bastardi, A., Sherman, D. K., Hsu, L., McGoey, M., & Ross, L. (2007). Bridging the partisan divide: Self-affirmation reduces ideological closed-mindedness and inflexibility in negotiation. Journal of Personality and Social Psychology, 93, 415–430.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Assocs.

Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, N.J.: L. Erlbaum Associates.

Corner, A., Whitmarsh, L., & Xenias, D. (2012). Uncertainty, scepticism and attitudes towards climate change: biased assimilation and attitude polarisation. Climatic Change, 114, 463–478.

Crawford, J. T. (2012). The ideologically objectionable premise model: Predicting biased political judgments on the left and right. Journal of Experimental Social Psychology, 48, 138–151.

Crawford, J. T., & Xhambazi, E. (in press). Predicting political biases against the Occupy Wall Street and Tea Party movements. Political Psychology.

Ditto, P. H., Pizarro, D. A., & Tannenbaum, D. (2009). Motivated moral reasoning. In H. R. Brian (Ed.), Psychology of learning and motivation (Vol. 50, pp. 307-338): Academic Press.

Downs, A. (1957). An economic theory of democracy. New York: Harper.

Druckman, J. N., & Bolsen, T. (2011). Framing, motivated reasoning, and opinions about emergent technologies. Journal of Communication, 61, 659-688.

Druckman, J., & Bolsen, T. (2012). How scientific evidence links attitudes to behaviors. In D. Dana (Ed.), The nanotechnology challenge (pp. 85-102). Cambridge, U.K.: Cambridge University Press.

Dunning, D. (1999). A newer look: motivated social cognition and the schematic representation of social concepts. Psychological Inquiry, 10, 1–11.

Dunning, D. (2003). The relation of self to social perception. In M. R. Leary & J. P. Tangney (Eds.), Handbook of self and identity (pp. 421–441). New York: Guilford Press..

Fernbach, P. M., Rogers, T., Fox, C. R., & Sloman, S. A. (2013). Political extremism is supported by an illusion of understanding. Psychological Science, 24, 939–946.

Feygina, I., Jost, J. T., & Goldsmith, R. E. (2010). System Justification, the Denial of Global Warming, and the Possibility of “System-Sanctioned Change”. Personality and Social Psychology Bulletin, 36, 326–338.

Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.

Gallup (2012, Jan. 12). Conservatives remain the largest ideological group in the U.S. Retrieved from http://www.gallup.com/poll/152021/conservatives-remain-largest-ideological-group.aspx.

Gastil, J., Braman, D., Kahan, D., & Slovic, P. (2011). The cultural orientation of mass political opinion. PS: Political Science & Politics, 44, 711–714.

Gervais, W. M., & Norenzayan, A. (2012). Analytic thinking promotes religious disbelief. Science, 336, 493–496.

Gigerenzer, G. (2000). Adaptive thinking : rationality in the real world. New York: Oxford University Press.

Giner-Sorolla, R., & Chaiken, S. (1997). Selective use of heuristic and systematic processing under defense motivation. Personality and Social Psychology Bulletin, 23, 84–97.

Gliem, J. and Gliem, R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Paper presented at the 2003 Midwest Research to Practice Conference in Adult, Continuing and Community Education, the Ohio State University, Columbus, Ohio. Retrieved from https://scholarworks.iupui.edu/bitstream/handle/1805/344/Gliem+&+Gliem.pdf?sequence=1.

Green, D. P., Palmquist, B., & Schickler, E. (2002). Partisan hearts and minds : political parties and the social identities of voters. New Haven, Conn.: Yale University Press.

Gusfield, J. R. (1993). The social symbolism of smoking and health. In R. L. Rabin & S. D. Sugarman (Eds.), Smoking policy: law, politics, and culture (pp. 49).

Haran, U., I., R., & Mellers, B. A. (2013). The role of actively open-minded thinking in information acquisition, accuracy and calibration. Judgment and Decision Making, 8, 188–201.

Hardin, G. (1968). The tragedy of the commons. Science, 162, 1243–1248.

Hastorf, A. H., & Cantril, H. (1954). They saw a game: A case study. The Journal of Abnormal and Social Psychology, 49, 129–134.

Hillman, A. L. (2010). Expressive behavior in economics and politics. European Journal of Political Economy, 26, 403–418.

Hoppe, E. I., & Kusterer, D. J. (2011). Behavioral biases and cognitive reflection. Economics Letters, 110, 97–100.

Iyer, R., Koleva, S., Graham, J., Ditto, P., & Haidt, J. (2012). Understanding libertarian morality: The psychological dispositions of self-identified libertarians. PLoS ONE, 7, e42366. http://dx.doi.org/10.1371/journal.pone.0042366.

Jacoby, W. G. (2000). Loess: A nonparametric, graphical tool for depicting relationships between variables. Electoral Studies, 19, 577–613.

Jaquet, Jennifer (2011). The pros and cons of Mechanical Turk for scientific surveys. Scientific American, Blogs. Retrieved from http://blogs.scientificamerican.com/guilty-planet/2011/07/07/the-pros-cons-of-amazon-mechanical-turk-for-scientific-surveys/.

Jenkins-Smith, H. C., Silva, C. L., Nowlin, M. C., & deLozier, G. (2011). Reversing nuclear opposition: Evolving public acceptance of a permanent nuclear waste disposal facility. Risk Analysis, 31, 629–644.

Jolls, C., & Sunstein, C. (2006). Debiasing through law. Journal of Legal Studies, 35, 199–242.

Jost, J. T., Glaser, J., Kruglanski, A. W., & Sulloway, F. J. (2003). Political conservatism as motivated social cognition. Psychological Bulletin, 129, 339–375.

Jost, J. T., Hennes, E. P., & Lavine, H. (in press). “Hot” political cognition: Its self-, group-, and system-serving purposes. In D. E. Carlson (Ed.), Oxford handbook of social cognition. New York: Oxford University Press.

Jost, J. T., Nosek, B. A., & Gosling, S. D. (2008). Ideology: Its resurgence in social, personality, and political psychology. Perspectives on Psychological Science, 3, 126–136.

Kahan, D.M. (2010). Fixing the communications failure. Nature, 463, 296–297.

Kahan, D.M. (2012a). Ideology, motivated reasoning, and cognitive reflection: an experimental study. Cultural Cognition Working Paper No. 107. Retrieved from http://ssrn.com/abstract=2182588.

Kahan, D.M. (2012b). Why we are poles apart on climate change. Nature, 488, 255.

Kahan, D. M., Hoffman, D. A., Braman, D., Evans, D., & Rachlinski, J. J. (2012). They saw a protest : Cognitive illiberalism and the speech-conduct distinction. Stanford Law Review, 64, 851–906.

Kahan, D. M., Jenkins-Smith, H., & Braman, D. (2011). Cultural cognition of scientific consensus. Journal of Risk Research, 14, 147–174.

Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L. L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change, 2, 732–735.

Kahan, D., Braman, D., Cohen, G., Gastil, J., & Slovic, P. (2010). Who fears the HPV vaccine, who doesn’t, and why? An experimental study of the mechanisms of cultural cognition. Law and Human Behavior, 34, 501–516.

Kahan, D. M., Slovic, P., Braman, D., & Gastil, J. (2006). Fear of democracy: A cultural critique of Sunstein on risk. Harvard Law Review, 119, 1071–1109.

Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral economics. American Economic Review, 93, 1449–1475.

King, G., Tomz, M., & Wittenberg., J. (2000). Making the most of statistical analyses: improving interpretation and presentation. American Journal of Political Science, 44, 347–361.

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.

Lavine, H., Johnston, C. D., & Steenbergen, M. R. (2012). The ambivalent partisan: How critical loyalty promotes democracy. New York, NY: Oxford University Press.

Leiserowitz, A. A. (2005). American risk perceptions: Is climate change dangerous? Risk Analysis, 25, 1433–1442.

Lessig, L. (1995). The regulation of social meaning. University of Chicago Law Review, 62, 943–1045.

Lewandowsky, S., Gignac, G. E., & Vaughan, S. (2012). The pivotal role of perceived scientific consensus in acceptance of science. Nature Climate Change, 3, 399–404.

Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2011). Individual differences in numeracy and cognitive reflection, with implications for biases and fallacies in probability judgment. Journal of Behavioral Decision Making, 25, 361–381.

Lilienfeld, S. O., Ammirati, R., & Landfield, K. (2009). Giving debiasing away: can psychological research on correcting cognitive errors promote human welfare? Perspectives on psychological science, 4, 390–398.

Liu, B. S., & Ditto, P. H. (2013). What dilemma? Moral evaluation shapes factual belief. Social Psychological and Personality Science, 4, 316–323.

Loewenstein, G. F., Weber, E. U., Hsee, C. K., & Welch, N. (2001). Risk as feelings. Psychological Bulletin, 127, 267–287.

Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: Effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098–2109.

Marx, S. M., Weber, E. U., Orlove, B. S., Leiserowitz, A., Krantz, D. H., Roncoli, C., & Phillips, J. (2007). Communication and mental processes: Experiential and analytic processing of uncertain climate information. Global Environmental Change-Human and Policy Dimensions, 17, 47–58.

Mooney, C. (2012). The Republican brain: The science of why they deny science—and reality. Hoboken, NJ: John Wiley & Sons.

Munro, G. D., & Ditto, P. H. (1997). Biased assimilation, attitude polarization, and affect in reactions to stereotype-relevant scientific information. Personality and Social Psychology Bulletin, 23, 636-653.

Nam, H. H., Jost, J. T., & Bavel, J. J. V. (2013). “Not for all the tea in china!” Political ideology and the avoidance of dissonance. PLoS ONE 8(4), 8, e59837 http://dx.doi.org/59810.51371/journal.pone.0059837.

Nisbet, M. C., & Scheufele, D. A. (2009). What’s next for science communication? Promising directions and lingering distractions. American Journal of Botany, 96, 1767–1778.

Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (2012). Analytic cognitive style predicts religious and paranormal belief. Cognition, 123, 335–346.

Pennycook, G., Cheyne, J., Barr, N., Koehler, D., & Fugelsang, J. (2013). Cognitive style and religiosity: The role of conflict detection. Memory & Cognition. Advance on-line publication. http://dx.doi.org/10.3758/s13421-013-0340-7.

Petrocik, J. R. (2009). Measuring party support: Leaners are not independents. Electoral Studies, 28, 562–572

Pew Research Center (2012, Oct. 15). More say there is solid evidence of global warming. Retrieved from http://www.people-press.org/files/legacy-pdf/10-15-12%20Global%20Warming%20Release.pdf.

Popper, K. R. (1959). The logic of scientific discovery. New York: Basic Books.

Richey, S,., & Taylor, B. (2012). How representatives are Amazon Mechanical Turk workers? The Monkey Cage, http://themonkeycage.org/2012/12/19/how-representative-are-amazon-mechanical-turk-workers/.

Rubin, D. B. (2004). Multiple imputation for nonresponse in surveys. Hoboken, N.J. ;: Wiley-Interscience.

Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science. Advance on-line publication. http://dx.doi.org/10.1177/2167702612469015.

Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology: General, 141, 423–428.

Sherman, D. K., & Cohen, G. L. (2002). Accepting threatening information: Self-affirmation and the reduction of defensive biases. Current Directions in Psychological Science, 11, 119–123.

Sherman, D. K., & Cohen, G. L. (2006). The psychology of self-defense: self-affirmation theory Advances in Experimental Social Psychology (Vol. 38, pp. 183-242): Academic Press.

Smith, E. R. (2000). Research Designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 17-39). Cambridge: Cambridge University Press.

Stanovich, K. E. (2011). Rationality and the reflective mind. New York: Oxford University Press.

Stanovich, K. E., West, R. F., & Toplak, M. E. (2011). Individual differences as essential components of heuristics and biases research. In J. S. B. T. Evans, K. I. Manktelow, D. E. Over & S. Elqayam (Eds.), The science of reason : a Festschrift for Jonathan St. B.T Evans (pp. 355-396). New York, NY: Psychology Press.

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23, 645–665.

Streiner, D. L. (2003). Unicorns Do Exist: A tutorial on “proving” the null hypothesis. Canadian Journal of Psychiatry, 48, 756–761.

Sunstein, C. (2003). Terrorism and probability neglect. Journal of Risk and Uncertainty, 26, 121–136.

Sunstein, C. (2006a). The availability heuristic, intuitive cost-benefit analysis, and climate change. Climatic Change, 77, 195–210. http://dx.doi.org/10.1007/s10584-006-9073-y.

Sunstein, C. (2006b). Misfearing: A reply. Harvard Law Review, 119, 1110-1125.

Sunstein, C. (2007). On the divergent American reactions to terrorism and climate change. Columbia Law Review, 107, 503–557.

Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50, 755-769. http://dx.doi.org/10.1111/j.1540-5907.2006.00214.x.

Toplak, M., West, R., & Stanovich, K. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39, 1275–1289.

United States. Public Health Service. (2000). Reducing tobacco use : a report of the Surgeon General Washington, D.C.: Dept. of Health and Human Services Supt. of Docs., U.S. G.P.O., distributor).

Weber, E. (2006). Experience-based and description-based perceptions of long-term risk: Why global warming does not scare us (yet). Climatic Change, 77, 103–120.

Weber, E. U., & Stern, P. C. (2011). Public understanding of climate change in the United States. American Psychologist, 66, 315–328. http://dx.doi.org/10.1037/a0023253.

Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C., Burns, W. J., & Peters, E. (2012). Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. Journal of Behavioral Decision Making, 26, 198–212.

Westen, D., Blagov, P. S., Harenski, K., Kilts, C., & Hamann, S. (2006). Neural bases of motivated reasoning: An fMRI study of emotional constraints on partisan political judgment in the 2004 U.S. presidential election. Journal of Cognitive Neuroscience, 18, 1947–1958.

Zaller, J. R. (1992). The nature and origins of mass opinion. Cambridge, England: Cambridge Univ. Press.7.

Appendix. Experimental stimuli

A. RANDOMLY ASSIGN to condition 1 or 2 or 3.

1. Control condition

Psychologists believe the questions you have just answered measure how reflective and open-minded someone is.

2. Skeptic-is-biased condition

Psychologists believe the questions you have just answered measure how reflective and open-minded someone is.

In one recent study, a researcher found that people who accept evidence of climate change tend to get more answers correct than those who reject evidence of climate change. If the test is a valid way to measure open-mindedness, that finding would imply that those who believe climate change is happening are more open-minded than those who are skeptical that climate change is happening.

3. Believer-is-biased condition

Psychologists believe the questions you have just answered measure how reflective and open-minded someone is.

In one recent study, a researcher found that people who reject evidence of climate change tend to get more answers correct than those who accept evidence of climate change. If the test is a valid way to measure open-mindedness, that finding would imply that those who are skeptical that climate change is happening are more open-minded than those who believe climate change is happening.

B. ALL CONDITIONS CRT_valid

How strongly do you agree or disagree with this statement? [strongly Disagree, moderately disagree, slightly disagree, slightly agree, moderately agree, strongly agree]

Yale Law School, P.O. Box 208215, New Haven, CT 06520. Email: dan.kahan@yale.edu.

Pennycook et al. (2012, p. 5) report that 53% of the subjects in their sample self-identified as liberal and 25% identified as conservative. Stratified national surveys suggest that approximately 20% of the general population self-identifies as liberal and 40% as conservative (Gallup, 2012).

The wording of the instructions supplied in each experimental condition is reproduced in the Appendix.

The experiment was first analyzed without imputed data, using listwise deletion for observations containing missing data (Kahan 2012a). The results did not differ materially from those reported.

To perform a more exacting test of whether the motivated-reasoning effect observed in the experiment was symmetric or asymmetric with respect to subjects’ ideological dispositions, a quadratic regression model was fit to the data. Formed by adding to Model 1 of Table 3 terms that squared the value of the subjects’ scores on Conserv_Repub, the resulting model reflected the assumption that the motivated-reasoning elicited by the experimental treatment would be stronger as subjects’ outlooks approached one extreme of the disposition measured by Conserv_Repub (either the “right-” “or the “left-wing” end) than it would be as that disposition approached the opposite extreme. The added terms had coefficients that were trivially different from zero and nonsignificant, and the addition of them did not result in a significant improvement in fit (Kahan 2012a).

In order for this result to support the inference that motivated reasoning is symmetric with respect to ideology in general, it must be assumed that that “liberal Democrats” and “conservative Republicans” have a comparable identity-protective stake in their positions on the particular issue on which subjects were exposed to counter-attitudinal evidence in this study. If, say, liberal Democrats are more intensely committed to being perceived as “open-minded” and “reflective” on climate change than conservative Republicans, then one might expect the former to feel more threatened by counter-attitudinal evidence and hence more pressured to dismiss such evidence in a reflexive or dogmatic fashion. The design of the study cannot rule this possibility out. But while such an explanation would “fit” the results of the study, it would not be particularly supportive of the “asymmetry thesis” premise (Jost et al., 2003; Nam et al., 2013) that conservatives are peculiarly vulnerable to ideologically motivated reasoning.

Item		% correct
WIDGETS	If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?	26%
BATBALL	A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?	13%
LILYPAD	In a lake, there is a patch of lilypads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?	25%

	Model 1	Model 2
Conserv_Repub	0.07 (1.45)	0.05 (0.97)
male		0.68 (6.60)
white		0.72 (5.41)
education		0.31 (8.39)
income		0.06 (3.32)
religiosity		−0.15 (−3.63)
LR χ²	2.1 (1 df)	261.9 (6 df)
Δ LR χ²		259.8 (5 df)

	Model 1	Model 2
Conserv_Repub	−0.12 (−1.60)	−0.12 (−1.58)
Skeptic-is-biased	−0.02 (−0.29)	−0.03 (−0.27)
Believer-is-biased	−0.18 (−1.74)	−0.17 (−1.63)
Con_Rep_x_skeptic	−0.50 (−4.63)	−0.50 (−4.62)
Con_Rep_x_believer	0.38 (3.59)	0.37 (3.47)
zCRT		0.04 (0.53)
C_R_x_zcrt		−0.05 (−0.76)
zCRT_x_skeptic		−0.01 (−0.12)
zCRT_x_believer		−0.07 (−0.65)
Con_Rep_x_zCRT_x_skeptic		−0.03 (−0.31)
Con_Rep_x_zCRT_x_believer		0.30 (2.95)
LR χ²	89.96 (5 df)	102.60 (11 df)
Δ LR χ²		12.64 (6 df)

Ideology, motivated reasoning, and cognitive reflection

Dan M. Kahan*

1 Introduction

2 Empirical and theoretical background

3 Study: sample, design, and hypotheses

3.1 Sample

3.2 Observational component

a. Measures.

b. Hypotheses.

3.3 Experimental component

a. Stimuli and measures.

b. Hypotheses.

3.4 Statistical power and missing data

4 Results

4.1 Observational component

4.2 Experimental component

a. Ideologically motivated reasoning generally.

b. Symmetry.

c. Interaction with CRT.

5 Discussion

5.1 Summary of results

5.2 Implications for ongoing study of dual-process and motivated reasoning

5.3 Implications for counteracting ideologically motivated reasoning

5.4 Conclusion: The tragedy of the science communications commons

References

Appendix. Experimental stimuli

A. RANDOMLY ASSIGN to condition 1 or 2 or 3.

1. Control condition

2. Skeptic-is-biased condition

3. Believer-is-biased condition

B. ALL CONDITIONS CRT_valid

Dan M. Kahan^*