Judgment and Decision Making, Vol. 13, No. 6, November 2018, pp. 639-651
Measurement is the core disgust problem: Response to Inbar and Scott (2018)Corey Cusimano* Edward B. Royzman# Robert F. Leeman$ Stephen Metas! |
Scott, Inbar and Rozin (2016) presented evidence that trait disgust predicts opposition to genetically modified food (GMF). Royzman, Cusimano, and Leeman (2017) argued that these authors did not appropriately measure trait disgust (disgust qua oral inhibition or OI) and that, once appropriately measured, the hypothesized association between disgust and GMF attitudes was not present. In their commentary, Inbar and Scott (2018) challenge our conclusions in several ways. In this response, we defend our conclusions by showing (a) that OI is psychometrically distinct from other affective categories, (b) that OI is widely held to be the criterial feature of disgust and (c) that we were well-justified to pair OI with the pathogen-linked vignettes that we used. Furthermore, we extend our critique to the new findings presented by Inbar and Scott (2018); we show that worry and suspicion (not disgust) are the dominant affective states one is likely to experience while thinking about GMF and that the true prevalence of disgust is about zero. We conclude by underscoring that the present argument and findings are a part of a larger body of evidence challenging any causal effect of disgust on morality.
Keywords: disgust, moral, GMO, methodology, trait disgust
Scott, Inbar and Rozin (2016) (hereafter, SIR) presented evidence that trait disgust underlies opposition to genetically modified foods (GMFs). We (Royzman, Cusimano & Leeman, 2017) disputed their claim on the basis that SIR did not appropriately measure trait disgust (disgust qua oral inhibition or OI)1 and that, once measured appropriately, it appears to play no role in people’s opposition to GMF. Instead we found that sensitivity to a potential/uncertain threat (as gauged by feelings of creepiness in response to a variety of “unpleasant” scenarios comprising the 7-item pathogen sub-scale [TDDS-P] [Tybur, Lieberman & Griskevicius, 2009]) was the best affective predictor of judgments regarding GMF. Inbar and Scott (2018) (hereafter, IS) have taken issue with our paper on many grounds. In most cases we believe their criticism misses the mark. In this paper, we lay out our rationale why and present additional empirical evidence that bears on the new findings that IS report. Though this response is a critique, we believe that Inbar, Rozin and Scott have raised some thought-provoking points and we would welcome the opportunity to explore them collaboratively in the future. We begin by reviewing our key objections to SIR and what we have done to address them.
SIR reported an association between subjects’ responses to the Disgust-Sensitivity Scale – Revised (DS-R, Olatunji et al., 2007), a refined variant of Haidt et al.’s (1994) trait disgust sensitivity scale, and their opposition toward GMF. This scale has been criticized for its content as well as its response options (e.g., Landy & Piazza, 2017; Tybur et al., 2009). Regarding the former, the majority of vignettes describe situations that could be interpreted as physically disgusting, but also creepy, irritating, frightening, surprising, or upsetting. Regarding the latter, only a subset of items (vignettes) ask participants to report how “disgusting” they find a situation or event, with the remaining items’ response options covering a wide range of negative states, including “bothered”, “upset”, and other non-specific expressions of avoidance. Exacerbating these issues, the meaning of “disgusting” itself has been has been shown to be broad enough to encompass a gamut of affective reactions (e.g., Landy & Piazza, 2017; Nabi, 2002; Royzman et al., 2017), making it the denotative equivalent of “bothered”. Given the multi-faceted nature of the DS-R, one would be hard pressed to see it as anything more specific than a measure of negative affect (with emphasis on creepiness, horror, disapproval, and disgust).
Our study was, in part, designed to ask if disgust sensitivity would still predict attitudes toward GMF when we could be more certain that we were measuring actual feelings of disgust. To address the limitations of the DS-R, we used a set of vignettes that are more pathogen-specific (TDDS-P, Tybur et al., 2009) as well as a 3-item measure of disgust as oral inhibition (OI), a measure derived from the theoretical meaning of disgust (Nabi, 2002; Royzman, Leeman & Sabini, 2008; see below). Prior studies have shown OI (unlike “disgust”2) to be specific to pathogen-linked content (Royzman, Atanasov, Landy, Parks & Gepty, 2014; Blake, Yih, Zhao, Sung & Harmon-Jones, 2016), unrelated to social desirability concerns (Royzman, Kim & Leeman, 2015), and resistant to figurative use (Royzman et al., 2008). We also included a set of concretely phrased non-disgust options that made it possible to isolate OI from some pertinent alternative reactions (e.g., social disapproval or unpleasant non-gustatory sensations). The inclusion of specific non-disgust options is a significant improvement for two reasons. First, including distinct alternative response options undercuts any motivation for a subject to use OI to express some other feeling that they may have or misreport what they feel in the interest of signaling that they have had some feelings at all. Second, it allows one to test whether any of these other affective reactions may have driven the original reported association between “disgust” and GMF attitudes. Our study was also an improvement in terms of scale administration (see Royzman et al., 2017, p. 467, for a summary).
With these refinements in place, the correlation between oral inhibition and people’s attitudes to GM-food was close to zero. A subsequent analysis showed that OI was not predictive of GMF attitudes either as a composite or at the level of its constituent responses (all ps> 0.4). Thus, contrary to the claims made by SIR, disgust-the-feeling (OI) appears to be unrelated to a desire to restrict GM-food (harms and benefits not withstanding). Notably, disgust-the-word (“disgusting”) still did rather well, serving as a significant predictor of opposition to GM-food (r = .25, p = .004), just as it did in SIR’s original study.
In spite of these deflationary results, Inbar and Scott make no attempt to defend their approach. Thus, it appears that what ultimately divides us is not the question of whether DS-R can be retained or salvaged as a measure of trait disgust but whether our approach offered enough of an improvement over SIR’s to make our findings more trustworthy.
IS advance four major reasons why they do not view our study as an improvement. First, they express a concern about our lack of proper psychometric analyses, thereby calling into question whether our novel high-granularity response options did in fact carve out distinct affective categories. Second, and most fundamentally, they assert that OI is incidental to the modern construal of disgust and therefore that we were unjustified using OI to operationalize disgust. Third, they argue that it is particularly inappropriate to pair OI with TDDS-P since it is characterized by lack of “oral” items. And fourth, they question the lay meaning of “creeped out” in the context of the TDDS-P vignettes. Finally, in their response, IS offer new evidence purporting to bolster their claim that disgust is the predominant feeling in reaction to thinking about GMF. We address each of these issues in turn.
One of Inbar and Scott’s central critiques is that we are not justified to assume that our scale of concrete, high-granularity options (e.g., “feel like gagging”) in fact measured separate affective categories (e.g., distinguishing oral inhibition from non-gustatory aversive sensations or moral disapproval).
In developing this critique, Inbar and Scott state early on that confirmatory factor analysis (CFA) is not appropriate for the three single-item factors in the measure because CFA requires a minimum of two (but preferably three) indicators for each latent factor (Brown, 2006). This is a valid point, but the issue is broader than that. Factor analysis in general (not only CFA but including exploratory factor analysis), is considered inappropriate for factors with less than two items (Costello and Osborne, 2005). As a result, the full 11-item measure utilized by Royzman et al. (2017) is not appropriate for inclusion in any type of factor analysis since it was designed to include multiple single-item factors. Thus, the exploratory factor analysis conducted and reported by Inbar and Scott was inappropriate on statistical grounds.
CFA is relevant to our measure because it was based on established scales, and items were grouped a priori according to presupposed factors based on theory, prior findings, and face validity (Brown 2006). Thus, for each of the seven vignettes we conducted CFA for the three putative factors that included at least two indicator variables. This yielded a total of 8 items per vignette: the three oral inhibition items, the three disapproval items, and the two epidermal discomfort items.
The model fit indices included (among others): chi-square (Hatcher, 1994), the SRMR (Hu & Bentler, 1999), the Comparative Fit Index (CFI; Bentler, 1990), and the Root Mean Square Error of Approximation (RMSEA; Brown & Cudeck, 1993).3 These indices indicated excellent fit of the data to the proposed model structure. For example, chi-square was statistically significant for 6 of the 7 vignettes with the “poop” vignette being the only one with a significant chi-square value. (Notably, chi-square is often significant even in cases when a model provides an acceptable fit to the data [Floyd & Widaman, 1995; Hatcher, 1994]). CFI values were greater than .95 for all seven vignettes and SRMR values were less than .08 for all seven vignettes. RMSEA was less than .08 for six vignettes with the other coming in under the .10 value that is considered acceptable. Additional indices were equally consistent with the model (contact authors for further details).
In summary, the single item factors included in the measure are not appropriate for any form of factor analysis. This is indeed an inherent limitation to the use of single item factors, however these items included in Royzman et al. (2017) were highly face valid. CFAs conducted with the remaining response options indicated excellent model fit for each of the seven vignettes measured. These results support a conclusion that the measure utilized in Royzman et al. (2017) was not psychometrically unsound as argued.
IS’s most pivotal critique is that there is a clear scientific consensus that oral inhibition is not a theoretically appropriate conceptualization of disgust. If they are correct, then our failure to observe an association between oral inhibition and GM attitudes has no significant bearing on their claim that feelings of disgust affect GM attitudes. However, it appears that IS are mistaken about the current consensus among disgust scholars.
From Darwin on, scholars viewed disgust as a category of food rejection (e.g., Rozin, Haidt & McCauley, 1993; Oluatunji & Sawchuck, 2005), whose origin is the “mammalian bitter taste rejection system” (Rozin, Haidt & Fincher, 2009, p. 1180), and whose facial emblem is the “gape” (Rozin, Lowery & Ebert, 1994) or the “sick face” (Widen, Pochedly, Pieloch & Russell, 2013). Rozin views disgust as “a food-related emotion” defined as “revulsion at the prospect of oral incorporation of offensive objects” (Rozin & Fallon, 1987, p. 23; Rozin, Haidt & McCauley, 2000) and affirms that all the classic disgust “scholars, as well as most others, posit a special relation between disgust and the mouth, and disgust and food rejection” (Rozin et al., 2000, p. 188). Haidt et al. (1994), whose trait disgust scale is the precursor to the DS-R (Olatunji et al., 2007), stipulate that disgust is “at its core, an oral defense” (p. 702) while Olatunji & Sawchuck describe it as a manner of food rejection and a “guardian of the mouth” (Oluatunji & Sawchuck, 2005, p. 935). Ekman (2003) explicitly linked disgust with gagging (see also Ekman, Matsumoto & Friesen, 2005). Thus, in an influential research report designed to unknot the more pointed theoretical meaning of disgust from the rougher lay meaning of the term, Nabi (2002) affirms that, though disgust’s broad “common usage … appears to reflect that which is not only repellent but also irritating or annoying” (Nabi, 2002, p. 695), “theoretically, disgust refers to the offence taken to noxious objects or ideas that evoke…nausea” (Nabi, 2002, p. 695) (see also Kayyal, Pochedly, McCarthy & Russell, 2015; Royzman & Sabini, 2001).
Most importantly, oral inhibition emerges as the criterial feature of disgust that contemporary scholars reach for when in need of a clear touchstone to separate disgust from non-disgust. Nabi (2002) used a desire to vomit (“feel like throwing up”) as a key criterion for establishing if the lay terms “disgusted/“disgust” brought to mind what was mainly disgust or some mix of annoyance and anger. In an analysis seeking to differentiate disgust from a related parasite avoidance adaptation, Kupfer and Fessler (2018) delineate the experience of being disgusted as involving “feelings of revulsion, nausea, gagging, the urge to vomit”. In arguing that Nazis are genuinely disgusting, Sherman, Haidt and Coan (2007) aimed to show that people watching a video about the neo-Nazi culture reported the tightening and clenching of their throats. In claiming that fraud and dishonesty provoke actual disgust, Chan, Boven, Andrade and Ariely (2014) sought to demonstrate that being exposed to instances of deceit reduces oral consumption of water or milk. And, as part of the project aiming to produce a highly discriminant facial display of disgust, one that would clearly differentiate it from anger, sadness, or fear, Widen et al. (2014) instructed an actress to imagine that she was feeling sick and was about to vomit (with the resultant “sick face” being highly similar to the “gape” face used by SIR).
Some scholars have also forewarned of the dangers of linking disgust with something as non-criterial as avoidance since, unlike OI, it would result in difficulties demarcating it from states such as “fear” (e.g., at spiders and gore; Cisler, Olatunji & Lohr, 2009; Woody & Teachman, 2000), which people also experience when exposed to pathogen vectors. In the case of a self-report measure such as the TDDS-P, operationalizing disgust as avoidance (e.g. “rate the degree to which this [bleeding cut] makes you want to move away”) would make it nearly impossible to separate it from a medley of other affective states (fear, creepiness, epidermal discomfort, disapproval) which TDDS-P is well capable of invoking (Landy & Piazza, 2017; Royzman et al., 2017).
The current scientific consensus is hard to miss. Oral inhibition is what scholars uniformly appeal to when the goal is isolate the criterial feature(s) of disgust, be it for the purpose of segregating the specific theoretical meaning from a broader lay one, differentiating disgust from a closely related non-disgust response, establishing that a non-prototypical disgust stimulus (e.g., Nazis, fraudsters) is genuinely disgusting, developing a discriminant non-verbal operationalization of disgust, or ascertaining cases of disgust-specific avoidance in a clinical setting. Any non-OI delineation of disgust is thus likely to be a minority position.
What about the authors that IS cite? Could they be the dissenting minority in question? The answer appears to be no. To the degree that these authors take a stab at delineating the disgust experience at all, all appear to endorse OI as the key component of that experience. Tybur et al. (2009) list “sensation of disgust” and “disgust facial expression” as the two primary features of the output of the “disgust system” (p. 69) and suggest that disgust is characteristically or “often accompanied by nausea, a desire to vomit, and a loss of appetite” (p. 70). Murray and Schaller (2016) limit their comment on the “emotional experience” of disgust to an observation that said experience has “evolved from a more ancient and functionally specific distaste response to oral stimuli (Rozin et. al, 2000), and many contemporary conceptual accounts identify infectious diseases as the primary selective pressure underlying its evolution” (p. 83). Finally, according to Curtis, de Barra, and Aunger (2011):
Contact with disgust elicitors, real or imagined, is associated with (i) a characteristic facial expression that is recognizable across cultures, (ii) behaviour patterns that include withdrawal, distancing, stopping or dropping the object of disgust and shuddering, (iii) physiological changes including lowered blood pressure and galvanic skin response, recruitment of serotonin pathways, increased immune strength, and (iv) reports of negative affect including nausea. (p. 390; italicized text is what Inbar and Scott quote as support for their view; bold emphasis is ours).
The quote merits two observations. First, Curtis et al.’s goal is clearly not to define or specify the criterial feature(s) of disgust, but rather to describe some of its observable identifiers or “associates”, none of which are at odds with the conception of disgust as OI. But more importantly, there is little doubt that, in the context of self-report measures, the identifier or associate of greatest relevance is (iv), which specifies what disgust is as an affective state and features nausea front and center.
None of this is obviously to dispute that disgust is linked to pathogen avoidance. Royzman and Sabini (2001) were among the earlier authors to characterize disgust as a system for minimizing toxin or pathogen exposure at the level of adaptive task description. However, they also noted that the aversive experience of OI is a good candidate response for motivating physical withdrawal. All in all, there is little doubt that the prevailing scientific consensus is squarely on the side of OI being the criterial feature of disgust. The authors that IS have appealed to are either a part of (or, at the very least, do not detract from) this broad consensus.
Even if OI comprised a defensible measure of disgust in general, it may not have been defensible in the context of TDDS-P. In IS’s view, OI is “a priori … a poor fit for these [TDDS-P] non-oral items” (Inbar & Scott, 2018, p. 5). They are not explicit about why this is an issue, but we have a hunch. When it comes to items featuring red sores, bleeding cuts, or scurrying insects, the key adaptive task seems to be that of avoiding a potential pathogen exposure by eschewing physical contact with its source. One could argue that gagging and pursing one’s lips would have little adaptive value; what one wants first and foremost is to put some distance between oneself and a potentially diseased entity. Thus, the argument goes, people may not have felt true oral inhibition in response to these items at all.
But if the major concern is a lack of variance due to a floor effect, this is a worry easy enough to dispel by looking at our data. We found that oral inhibition (tied with creeped out) was the strongest high-granularity response reported in reaction to the TDDS-P items (with OI’s coefficient of variance [0.42] being 20% higher than that of “disgusted” [0.23], which did predict resistance to GMF, in line with Scott et al. [2016]). We see no reason why this should be otherwise. Other than the “moldy leftovers” item and the scurrying cockroach item (see van Huis, 2017 on the evidence of insect consumption among early humans and through evolutionary history), at least three items in the TDDS-P feature explicit (body odor) or implied (sweaty palms, dog poop) malodorousness, one of the key inputs to the food rejection system (Rozin et al., 1994)4. This issue to the side, it is unclear why the “orality” of an item should matter in the first place. Congruent with the theory of preadaptation (i.e., the co-option of existing structures/systems for new use; Rozin & Fallon, 1987), there is plentiful evidence that oral inhibition is elicited by a range of “non-oral” items ranging from bodily fluids (Royzman et al., 2008, Pilot) and injuries (Chapman, Kim, Susskind & Anderson, 2009; Royzman et al., 2008, Pilot) to anomalous sex (Royzman et al., 2008; Studies 1 and 2; Royzman et al., 2014). In sum, our use of the TDDS-P scale for eliciting oral inhibition is as well justified as the use of OI itself.
Let’s collect our results. IS critiqued OI as a theoretically appropriate way to conceptualize and operationalize disgust while also contending that it was a particularly poor match for the TDDS-P vignettes. Having reviewed IS’s arguments, we have concluded that (1) according to the prevailing scientific consensus, oral inhibition is the criterial feature of disgust, (2) has no clear alternative, and (3) is the only reliable way to demarcate disgust from other pathogen avoidance mechanisms (creepiness, epidermal discomfort, fear), at least in the context of a self-report measure à la TDDS-P. Furthermore, (4) the combined weight of theory and evidence strongly indicates that our use of the TDDS-P scale was appropriate for eliciting oral inhibition. Along with the psychometric evidence in the section above, our decision to use a wide range of additional affective response options (including some face valid single-factor response options) further engenders confidence that subjects used our measure as intended.
What can be said about the overall validity of our measure? In modern validity theory (Newton & Shaw, 2013), validity is not considered a property of the test, but of a particular interpretation of that test adopted for a given purpose and whether this interpretation is justified in light of the totality of evidence before us. The key interpretative move that undergirds all conclusions in Royzman et al. (2017) is that, imperfect as our assessment might have been, it is a considerably better proxy for the feelings of disgust (as understood by the scientific community at large) and their relationship to GMF attitudes than SIR’s. Everything that has been said above — OI’s hard-to-argue status as the criterial feature of disgust, its apparent specificity, its psychometric differentiation from Disapproval and Epidermal Discomfort, its selective tracking of pathogen-linked content — makes the move well justified.
If disgust sensitivity, properly measured, does not predict GMF attitudes, why might disgust sensitivity, improperly measured, do so? We hypothesized that sensitivity to potential threats might have, among other things, confounded the relationship between improperly-measured disgust sensitivity and GMF attitudes. This hypothesis was supported by a correlation between GMF attitudes and feeling “creeped out” in response to the TDDS-P.
IS claim that, while “creeped out” may be associated with unease about a potential threat or risk5, it is an open question if this is the meaning that subjects had in mind when rating the items of TDDS-P. Implied in this worry is the claim that (a) the canonical elicitors of creepiness (those that one would expect to loom large in a subject’s minds when queried about the meaning of “creepiness” as such) are categorically different from (b) the content of the TDDS-P vignettes. But is this borne out by the evidence? McAndrew and Koehnke (2016) conclude from their study that non-normative health and hygiene (e.g., being extremely thin or obese, having greasy hair or pale skin) are typical triggers of creepiness (see also Watt, Maitland, & Gallagher, 2017, for a convergent analysis). In light of this, it appears that the characteristic triggers of feeling creeped out are well represented in TDDS-P. Indeed, half of the TDDS-P items portray individuals with non-normative, unhealthy physical characteristics or poor hygiene, including “Sitting next to someone who has red sores on their arm.”, “Standing close to a person who has body odor.” “Shaking hands with a stranger with sweaty palms”, and “Accidentally touching a person’s bloody cut”.
Whatever “creeped out” may or may not mean in the context of TDDS-P, it clearly not a stand-in for oral inhibition, the construct that comprises the theoretical meaning of disgust. It is this construct that SIR and IS presumably would want to implicate, not merely as a trait predictor, but also as a state antecedent, of people’s avowed opposition to genetically engineering animals and plants. Thus, the strong correlation between “creeped out” and GMF attitudes should not be taken as evidence in favor of IS’s thesis.
In the penultimate section of their commentary, Inbar and Scott present an empirical critique of what they perceive to be our theoretical counterpoint to their “disgust fuels the moral opposition” proposal, i.e., that “fear, not disgust, is the primary emotion associated with GM food opposition” (Inbar & Scott, 2018, p. 3). They offer no textual evidence for this claim. This is not surprising since we never made such a claim. Our paper was concerned with the experience of “creepiness” (which we explicitly and repeatedly set apart from that of “fear” both in the principal study and in the supplementary survey) as a proxy for trait sensitivity to a potential hazard or risk. Our claim that sensitivity or vigilance towards risks may underlie GMF opposition is not equivalent to the claim that fear is the dominant reaction to thinking about GMFs, nor that fear (or a related feeling) is a causal antecedent of GMF opposition6. Therefore, IS are largely correct that we do not show evidence in favor of the claim that state fear is the dominant reaction to GMFs, but this is because no such evidence has ever been searched for in the first place.
This misunderstanding to the side, we have no principled opposition to the view that disgust is the dominant affective reaction towards GMF. That said, we were skeptical of the evidence that IS and SIR presented defending this claim, and conducted a study to re-assess the prevalence of disgust as a reaction to GMF.
We conducted a study (described in detail in the Appendix) geared to re-reassess the absolute and relative prevalence of state disgust when thinking about GMF. Both SIR and IS presented some studies to this effect. Our study departed from theirs in several ways. First, rather than presenting subjects with GMF-related vignettes, (e.g., unintentionally eating genetically modified salmon), subjects were asked simply to report what they felt while thinking about GMF. This affords us a more ecologically valid measure of the types of stimuli/vignettes that might naturally come to mind when thinking about whether GMF should be opposed or prohibited.
We also modified the way we probed subjects’ affective states. SIR asked their subjects to report which of the two response options (disgust vs. anger) best reflects their reaction to different GM scenarios. IS used a similar method with three response options (disgust, anger, and fear) instead. However, this method comes with some strong assumptions. First, it forces people to report one feeling or another (leaving no room for the possibility that no emotion is being felt). Second, it operationalizes disgust as the word “disgust” (assuming, once again, a close correspondence between the theoretical and lay meanings of the term, an assumption disconfirmed some 15 years ago; see also our arguments above). Third, it limits the range of non-disgust options to one (“anger” in SIR) or two (“fear” and “anger” in IS), assuming that these states would adequately and fairly cover anything and everything else one could feel in this case. These assumptions are difficult to defend, and therefore call into question the validity of subjects responses7. To address this, in addition to running a conceptual replication of SIR’s study, we designed two additional conditions that do not make these assumptions and so furnish a truer test of what feelings, if any, people experience.
Table 1: Proportion of subjects choosing/using disgust-related terms as a function of study condition and evaluative orientation (absolute opposition, non-absolute opposition, non-opposition).
Condition 2-emotion FC 12- emotion FC OERM Note. FC = forced choice; OERM = open-ended response mode. Superscripts indicate whether disgust is more dominant than (a), is dominated by (b), or is equivalent to (c) non-disgust by Chi-square goodness-of-fit at α = 0.05.
At the beginning of the study, all subjects (N = 1048 recruited, N = 922 retained following attention check) were queried about whether they opposed or did not oppose “genetically engineering plants and animals for food production” (thus separating opponents from non-opponents) and whether they thought it “should be prohibited no matter how great the benefits and minor the risks of allowing it” (with an affirmative answer leading to the subject being classified as an opposing absolutist). Then, after completing a brief attention check, subjects were randomly assigned to one of three affective report conditions: (1) a replication of IS’s study in which subjects were forced to choose between two feeling states, Angry or Disgusted (2-item FC), (2) a 12-item forced choice condition (12-item FC) in which subjects were forced to choose between feeling Afraid, Angry, Bothered, Confused, Creeped out, Excited, Frustrated, Grossed out [a slang term shown to correspond more precisely than “disgusted” to the theoretical meaning of disgust, Nabi, 2002], Neutral, Sad, Suspicious, or Worried, or (3) an open-ended response mode condition (OERM).
The OERM condition was unique in that it left open the possibility that people do not feel or perceive themselves as having affective reactions at all. Rather, in this condition, subjects were prompted to “use the space below to describe (as you would in a diary) what is going though your mind and what you are experiencing as you are thinking about the practice of genetically engineering plants and animals for food production”. Subjects were further enjoined to note that there are “no right or wrong answers” and not to “hold back or self-censor in any way”.
To diagnose the prevalence of affective language in OERM, we carried out a series of content analyses using a total of 81 affective word stems corresponding to the feeling concepts tapped in the 12-item FC condition of the study. 80 of these were organized into 8-item clusters detailed in Table 2 (Appendix). Due to its precise classification being the present point of contention, “creeped out” was retained as a “free-standing” term, though, in our view, the evidence from prior research (Royzman et al., 2017; McAndrew & Koehnke, 2016), clearly substantiates its inclusion in a Fear (Afraid + Worry + Suspicion) super-cluster.
As we predicted, SIR’s procedure for measuring state disgust overestimates the absolute prevalence of disgust in relation to other, more assumption-free methods (see Table 1 above). When subjects’ choices were limited to disgust and anger, disgust substantially and significantly dominated non-disgust for absolute opponents (73.3 percent) as well as for the sample as a whole (62.2 percent). However, only 2.8% of the sample chose “grossed out” in the 12-item FC condition (Table 3, Appendix) and only 1.7% of the sample mentioned one of the target disgust-related words in the OERM condition (Table 4, Appendix), both significant drops from the 62.2 percent, χ2(1, 633) = 256.55, p< .0001 and χ2(1, 598) = 246.5, p< .0001, respectively (based on the “N-1” Chi-squared test as recommended by Campbell, 2007, and Richardson, 2011), with no significant difference between the 12-item condition and OERM (χ2[1, 633] = 0.82, p = 0.36).
The open-ended response format (Table 4, Appendix), which provides the most assumption-free approach to the problem, suggests that very few people spontaneously perceived themselves as experiencing any emotion at all (41 out of 288, 14.2%). For those apt to experience any feeling at all, feeling worried appears to be three times more common than feeling disgusted (see Appendix for details). Finally, it is worth noting that 1.7 percent (the absolute prevalence of disgust in the open-ended condition) is likely to be an overestimate and that a truly conservative estimate of spontaneously experienced and causally relevant state disgust is likely to be closer to zero (see Appendix for discussion).
These results strongly disqualify disgust as the driver of absolutist opposition in the face of consequentialist reasons. If Scott et al.’s (2016) sentimentalist viewpoint was correct, and absolute opposition to GM food was in fact “disgust-based” (p. 316) or “disgust-backed” (p. 322), the last thing one would expect to find is that the majority of GMF’s absolutist opponents feel no disgust at all.
In this reply, we defended our critique of the relationship between trait disgust and GMF and extended it to the issue of state disgust. In both cases, SIR and IS appear to overestimate the role that disgust may play in opposing GMF due to imprecisely measuring disgust or by providing limited response options that artificially constrain a subject’s capacity to state their feelings (or lack thereof). When these limitations are addressed, we find a different suite of feelings to come forth, by and large, related to the perceptions of danger, misfortune, or risk.
When considered in the broader context of disgust and moral judgment research, our findings are not at all surprising. A meta-analysis (Landy & Goodwin, 2015), and large-scale replication (Johnson, Cheung & Donnellan, 2014) both conclude that feelings of incidental disgust have practically no influence on moral judgment. Similar in spirit are several past studies (e.g., Fessler, Arguello, Mekdara & Macias, 2003; Royzman, Leeman & Baron, 2009; see also Royzman, Kim & Leeman, 2015) that found no significant association between disgust sensitivity and moral evaluations when special design precautions (time delay, misdirection, appropriate controls) were in place. Our own (Royzman et al., 2017) study found no relationship between the fine-tuned measure of disgust as OI and either GMF or moral purity concerns (as measured by MFQ-P, Graham et al., 2011]; r = .14, p = .09).8 The findings assembled in recent years have lead another research group to conclude that “there is little evidence for a unique predictive effect of physical, pathogen-based disgust experiences on moral judgments” (Piazza, Landy, Chakroff, Young & Wasserman, 2018, p. 75). From the Bayesian point of view, one’s a priori expectation that disgust predicts GMF moralization should be fairly low.
By contrast, our expectation that people will be principally worried or suspicious about the consequences of GM food should be high. Many studies have provided evidence that people critique GM food based on their unease about the possible harm that genetically modified food may have on the populace, and express a suspicion that scientists have not studied GM sufficiently or long enough to rule out negative effects (e.g., Chen & Li, 2007; Connor & Siegrist, 2010; Onyango & Nayga, 2004; Prati, Pietrantoni & Zani, 2012; Rzymski & Krolczyk, 2016). Indeed, a review of subjects’ verbatim statements in the OERM condition of our study, indicates that 60 percent of the absolutist opponents and 65 percent of all opponents were thinking uniquely in terms of harm and risk.
All in all, our study (Royzman, et al., 2017) was methodologically and conceptually sound. Our critique, as well as our novel findings, fit neatly with the recent pattern of findings showing that disgust plays little role in moral judgment. The burden of proof is on those who wish to claim otherwise.
We recruited 1048 individuals from Amazon’s Mechanical Turk (561 reported Female, AgeM = 35.7 AgeSD = 12.23).
At the beginning of the study, all subjects were queried about whether they opposed or did not oppose “genetically engineering plants and animals for food production” (thus separating opponents from non-opponents) and whether they thought it “should be prohibited no matter how great the benefits and minor the risks of allowing it” (with the affirmative answer leading to the subject being classified as an opposing absolutist). This allowed us to sort each subject in one of the three categories: absolutist opponent, non-absolutist opponent, non-opponent. Then, after completing a brief attention check, subjects were randomly assigned to one of the three affect report conditions described below: (1) a two-item forced choice condition (2-item FC) serving as a replication of the study reported in SIR, (2) a 12-item forced choice condition (12-item FC), or (3) an open-ended response condition (OERM).
In the 2-item FC (modeled after Scott et al., 2016) and 12-item FC conditions, subjects were asked to indicate which of the available affect terms best described what they felt while thinking about genetically engineering plants and animals for food production. In 2-item FC, the options were limited to Angry and Disgusted. In the 12-item condition, the options were considerably broader: Afraid, Angry, Bothered, Confused, Creeped out, Excited, Frustrated, Grossed out9, Neutral, Sad, Suspicious, and Worried.
We reasoned that “worry”, “suspicion”, and other aforementioned terms would constitute better alternatives to disgust than “anger” (or “fear”). The bulk of the published data on GMF attitudes shows that people are concerned about the possible harmful consequences that these products may have on their well-being or health, and are suspicious that scientists have studied GM sufficiently or long enough to rule out key negative effects (e.g., Chen & Li, 2007; Connor & Siegrist, 2010; Onyango & Nayga, 2004; Prati, Pietrantoni & Zani, 2012; Rzymski & Krolczyk, 2016). And, as pointed out by Jonathan Baron (personal communication, June 15, 2018), “worry” is also the affective category most commonly cited in Slovic’s “psychometric approach to risk” studies of lay responses to emerging technologies and medicine (e.g., Peters, Slovic, Hibbard & Tussler, 2006; Slovic, 2000). Consistent with Slovic, we see worry largely as the ruminative cousin of fear, arising as part of the affective sequelae of prospecting (Seligman, Railton, Baumeister & Sripada, 2013) future threats over which one can exert some (but only partial) control.
The two forced choice conditions were alike in that all the subjects had to feel something, and it was up to us, the researchers, what feelings they could express. By contrast, in the third open-ended response mode (OERM) condition, we left open the possibility that people do not feel anything at all. Rather, in this condition, subjects were prompted to “… use the space below to describe (as you would in a diary) what is going though your mind and what you are experiencing as you are thinking about the practice of genetically engineering plants and animals for food production.”
Table 2: 81-word affective word stems used in the OERM condition of the study.
Category Word Stems Disgust Disgust, Gag, Gross, Nausea, Repuls-, Revolt, Revul-, Sick Angry Anger, Fum-, Fur-, Indigna-, Irritate, Mad, Piss, Rag- Confused Baffl-, Bemus-, Bewilder-, Confound, Confus-, Mystif-, Perplex-, Puzzl- Excited Animat-, Eager, Energ-, Enthusiast-, Exhilarat-, Interest, Passion, Thrill Afraid Afraid, Dread, Alarm, Fear, Fright, Horr-, Scar-, Terr- Frustrated Discontent, Discourag-, Displeas-, Dissatisfy, Exasperat-, Frustrat-, Unfulfill-, Unsatisf- Neutral Apath-, Cold, Cool, Emotionless, Indifferent, Neutral, Unconcern, Uninterest Sad Depress-, Down, Gloom, Grief, Melancholy, Sad, Sorrow, Unhappy Suspicious Suspicio-, Skeptic-, Cautio-, Wary, Doubt, Hesitat-, Distrust, Misgiving Worry Anxi-, Concern, Disturb, Unease, Nerv-, Stress, Trouble, Worr- Bothered Bother Note. In all cases, subjects either used a single affective term or two affective terms from the same cluster, making all responses assignable to a single cluster.
The subjects were further enjoined to note that there are “no right or wrong answers” and not to “hold back or self-censor in any way”10. This circumlocution was motivated by several prior reports (reviewed by Feldman Barrett, 2017) suggesting that framing an affect-related question in terms of specific emotion concepts or feelings may result in an overestimate of a particular affective state due to subjects’ constructing their internal experiences around the concepts in question. Thus, the OERM condition was the most conservative (assumption-free) of the three experimental conditions in that it made no direct reference to affect.
To diagnose the prevalence of affective language in OERM, we carried out a series of content analyses using a total of 81 word stems corresponding to the feeling concepts tapped in the 12-item FC condition of the study. 80 of these were organized into 8 item-clusters detailed in Table 2.11 (No affective cluster was formed around “bothered” due the extremely low granularity of the term.) Due to its precise classification being the present point of contention, “creeped out” was retained as a “free-standing” term, though, in our view, the evidence from prior research (Royzman et al., 2017; McAndrew & Koehnke, 2016), clearly substantiates its inclusion in the Fear (Afraid + Worry + Suspicion) super-cluster.
124 subjects (12% of our entire sample) failed a brief attention check and were removed. Including these individuals has no noticeable effect on the results. All analyses reported below were performed on the remaining 922 subjects. Of these subjects, 424 (46%) indicated that they opposed GMF. Of people who indicated that they currently oppose GMF, 312 (74%) agreed with the statement that GMFs should be prohibited “no matter what”, and thus would be considered as “absolutist” opposers by SIR’s criterion.
Table 3: Frequency of feeling states selected in the 12-emotion forced choice condition.
Emotion Neutral Suspicious Worried Excited Bothered Confused Angry Creeped out Grossed out Afraid Frustrated Sad Note. Rows ordered by rank frequency in total sample. “Grossed out” highlighted to indicate rank.
As we predicted, SIR’s procedure for measuring state disgust overestimates the absolute prevalence of disgust. When subjects’ choices were limited to disgust and anger, disgust substantially and significantly dominated non-disgust for absolute opponents (73.3 percent) as well as for the sample as a whole (62.2 percent; see Table 1, main text). By contrast, only 2.8% of the sample chose “grossed out” in the 12-item FC condition (Table 3), and only 1.7% of the sample mentioned one of the target disgust-related words in the OERM condition (Table 4), both significant drops from 62.2 percent, χ2(1, 633) = 256.55, p< .0001 and χ2(1, 598) = 246.5, p< .0001, respectively (based on the “N-1” Chi-squared test as recommended by Campbell, 2007, and Richardson, 2011]), with no significant difference between the 12-item condition and OERM (χ2[1, 633] = 0.82, p = 0.36).
Table 4: Frequency of feeling states selected in the open response choice format condition.
Emotion Absolutely Oppose (N = 98) Total Sample (N = 288) % Emotion Terms (N = 38) Worry 3.3% 5.9% 44.7% Suspicious 2.2% 2.1% 15.8% Afraid 3.3% 1.7% 13.2% Disgust 4.3% 1.7% 13.2% Angry 2.2% 0.7% 5.3% Sad 1.1% 0.7% 5.3% Creeped Out 0.0% 0.7% 5.3% Neutral 0.0% 0.3% 2.6% Bother 0.0% 0.0% 0.0% Note. Rows ordered by rank frequency in total sample. Disgust highlighted to indicate rank.
Disgust does not appear to be an especially prominent feeling for people absolutely opposed to GMF, either. Only 6.7% of these subjects selected “grossed out” and only 4.3% of these subjects provided one of the disgust-cluster words in OERM (with 4 out of 5 cases using the non-specific word “disgust”, which commonly means merely “disapproval”). These both represent substantial decreases from SIR’s 2-condition method,χ2(1, 220) = 99.51, p< .0001 and χ2(1, 208) = 99.31, p< .0001, respectively (with no significant difference between 12-item FC and OERM, χ2[1, 196] = 0.53, p = 0.46). In contrast to disgust, other feelings, e.g., worry, suspicion, and apathy were far more common in people’s reactions. The open-ended response format, which provides the most assumption-free approach to the problem, suggests that very few people spontaneously perceived themselves as experiencing any emotion at all (41 out of 288, 14.2%), though opponents appeared somewhat more likely to do so (24/140, 17%) than non-opponents (14/148, 9.5%), χ2(1, 288) = 3.07, p = .08.
For those apt to experience any feeling at all, feeling worried appears to be three times more common than feeling disgusted (Table 4), though even worry comprises a relatively small proportion of the overall sample (5.9% for worry vs 1.7% for disgust). The sheer paucity of affect under OERM is in striking contrast to the conclusions one might have drawn from the 2-item FC. Finally, it is worth noting that 1.7 percent (the absolute prevalence of disgust in the open-ended condition) is likely to be an overestimate. As indicated earlier, in everyday speech, “disgust” or “disgusting” are often used to signify “immoral”, “bothered”, or “disliked”. This also likely overestimates the extent to which these feelings would be causally implicated in (or be even temporally antecedent to) the rise of absolutist disapproval. Thus, a truly conservative estimate of spontaneously experienced and causally relevant state disgust (disgust quo OI) is likely to be closer to zero.
There is little doubt that IS/SIR would challenge these results. They could, for example, argue that their 2016 assessment was not predicated solely on emotion words. In their original study (Scott et al., 2016), some subjects indicated their preference by being asked to select from a set of facial displays: two anger faces (the “maximal” anger face in the open mouth form with bilateral lip raise and the “minimal” anger face in the lip press form) and two disgust faces (the “maximal” gape/tongue protrusion disgust face and the “minimal” bilateral upper lip raise disgust face).
There are a number of problems with this method – key among them is the inclusion of the minimal disgust face as one of the bases for demarcating disgust from anger. As Rozin et al. (1994) themselves have observed,
“Among disgust situations, upper lip raise [AU 10] was the significant (p < .01) predominant response in two cases: ‘seeing a deformed person’ and ‘looking at pictures of the slaughter at a WWII concentration camp’…Upper lip raise was also the predominant response for anger, contempt, and rights violation (stealing), confirming its role in the anger expression. Thus…the major results of the choice test… affirm a link between upper lip raise and other moral emotions.” (p. 878).
According to Rozin et al.’s own data (see also Widen et al., 2013), the upper lip raise AU was not selectively related to the experience of disgust. Thus, it appears that, as with “disgust” itself, one of the constituent disgust displays lacks discriminant validity as a gauge of disgust vs. anger, contempt, and disapproval as well as any capacity to differentiate between physical and socio-moral domains.
To address this concern, we re-analyzed Scott et al.’s study comparing the maximal (gape) disgust face with its maximal anger counterpart.12 With the minimal disgust and anger faces removed, we no longer observed significant anger/disgust differences for any of the four scenarios. This was the case irrespective of whether we looked at the sample as a whole or limited our analyses to GMF opponents only.
One could also argue that forcing subjects to choose between two types of facial configurations that visually stereotype two folk emotion concepts inherits all or most of the problems we discussed above in the context of emotion words. These include the key criticism that applied to the forced-choice approach: people are forced to “feel” something (leaving no room for the hypothesis that they process the information in an emotion-free way). And the most likely emotions, including suspicion, worry, or apathy, are clearly not on the list. The face of fear is fundamentally a face of terror and there is simply no established facial configuration for the lower-activation states of “worry”, “suspicion”, or “concern”.
Finally, because Inbar and Scott’s presentation of their newer findings emphasize faces over words, it is worth noting that many of the theoretical and methodological assumptions underlying this method have been strongly and repeatedly questioned (see Russell, 1994, 2003; Feldman Barrett, 2017 for discussion). In our view, the use of facial displays is still of historical interest, but its current scientific status is so unsettled that its use as a means to resolve yet another scientific debate is contentious at best.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.
Blake, K. R., Yih, J., Zhao, K., Sung, B., & Harmon-Jones, C. (2016). Skin-transmitted pathogens and the heebie jeebies: evidence for a subclass of disgust stimuli that evoke a qualitatively unique emotional response. Cognition and Emotion, 31(6), 1153–1168.
Brown, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen, & J. S. Long (Eds.), Testing Structural Equation Models (pp. 136–162). Newbury Park, CA: Sage.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY, US: Guilford Press.
Campbell, I. (2007) Chi-squared and fisher-irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine, 26(19), 3661–3675.
Chan, C., Boven, L. V., Andrade, E. B., & Ariely, D. (2014). Moral violations reduce oral consumption. Journal of Consumer Psychology, 24(3). 381–386.
Chapman, H. A., Kim, D. A., Susskind, J. M., & Anderson, A. K. (2009). In bad taste: evidence for the oral origins of moral disgust. Science, 323(5918), 1222–1226.
Chen, M., & Li, H. (2007). The consumer’s attitude toward genetically modified foods in taiwan. Food Quality and Preference, 18(4), 662–674.
Cisler, J. M., Olatunji, B. O., & Lohr, J. M. (2009). Disgust sensitivity and emotion regulation potentiate the effect of disgust propensity on spider fear, blood-injection-injury fear, and contamination fear. Journal of Behavior Therapy and Experimental Psychiatry, 40(2), 219–229.
Connor, M., & Siegrist, M. (2010), Factors influencing people’s acceptance of gene technology: the role of knowledge, health expectations, naturalness, and social trust. Science Communication, 32(4), 514–538.
Costello, A. B., & Osborne J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 173–178.
Curtis, V., de Barra, M., & Aunger, R. (2011). Disgust as an adaptive system for disease avoidance behaviour. Philosophical Transactions of the Royal Society B: Biological Sciences, 366(1563), 389–401.
Ekman P. (2003). Emotions revealed: Recognizing faces and feelings to improve communication and emotional life. New York, NY: Henry Holt.
Ekman, P., Matsumoto, D., & Friesen, W. V. (2005). Facial expression in affective disorders. In P. Ekman & E. L. Rosenberg (Eds.), Series in affective science. What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.). (pp. 429–440). New York, NY, US: Oxford University Press.
Feldman Barrett, L. How emotions are made: The secret life of the brain. (2017). New York, New York: Houghon Mifflin Harcourt Publishing Company.
Fessler, D. M., Arguello, A. P., Mekdara, J. M., & Macias, R. (2003). Disgust sensitivity and meat consumption: A test of an emotivist account of moral vegetarianism. Appetite, 41(1), 31- 41.
Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299.
Graham J., Nosek, B. A., Haidt, J. Iyer, R., Spassena, K., Ditto, P. H. (2011). Mapping the moral domain. Journal of Personality and Social Psychology, 101(2), 366–385.
Haidt, J., McCauley, C., & Rozin, P. (1994). Individual differences in sensitivity to disgust: A scale sampling seven domains of disgust elicitors. Personality and Individual Differences, 16(5), 701–713.
Hatcher, L. (1994). A step-by-step approach to using the SAS system for factor analysis and structural equation modeling. Cary, NC: The SAS Institute.
Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 1–55.
Inbar, Y. & Scott, S. E. (2018). (in press). People respond to gm food with disgust more than fear: Comment on Royzman, Cusimano, and Leeman (2017). Judgment and Decision Making, 13.
Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does cleanliness influence moral judgments? A direct replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45(3), 209–215.
Kayyal, M. H., Pochedly, J., McCarthy, A., & Russell, J. A. (2015). On the limits of the relation of disgust to judgments of immorality. Frontiers in Psychology, 6, 951.
Kupfer, T. & Fessler, D. (2018). Ectoparasite defence in humans: Relationships to pathogen avoidance, and clinical implications. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1751).
Landy, J. F., & Goodwin, G. P. (2015). Does incidental disgust amplify moral judgment? A meta- analytic review of experimental evidence. Perspectives on Psychological Science, 10(4), 518–536.
Landy, J. F., & Piazza, J. (in press). Reevaluating moral disgust: sensitivity to many affective states predicts extremity in many evaluative judgments. Social Psychological and Personality Science.
McAndrew, F. T., & Koehnke, S. S. (2016). On the nature of creepiness. New ideas in psychology, 43, 10–15
McDonald, R. P., & Ho, M. H. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82.
Muth, E. R., Stern, R. M., & Koch, K. L. (1998). A psychophysiological paradigm for the study of nausea: A rotating optokinectic drum, the electrogastrogram, and the nausea profile. Journal of Psychophysiology, 12, 56–63.
Murray, D. R., & Schaller, M. (2016). The behavioral immune system: Implications for social cognition, social interaction, and social influence. In S, Mahfould (Eds.), Advances in Experimental Social Psychology, 53, 75–129.
Nabi, R. L. (2002). The theoretical versus the lay meaning of disgust: implications for emotion research. Cognition and Emotion, 16(5), 695–703.
Newton, P. E., & Shaw, S. D. (2013). Standards for talking and thinking about validity. Psychological Methods, 18(3), 301–319.
Olatunji, B. O., Sawchuk, C. N. (2005). Disgust: Characteristic features, social manifestations, and clinical implications. Journal of Social and Clinical Psychology, 24(7), 932–962.
Olatunji, B. O., Williams, N. L., Tolin, D. F., Abramowitz, J. S., Sawchuk, C. N., Lohr, J. M., & Elwood, L. S. (2007). The disgust scale: item analysis, factor structure, and suggestions for refinement. Psychological Assessment, 19(3), 281–297.
Onyango, B., & Nayga, R. (2004). Consumer acceptance of nutritionally enhanced genetically modified food: relevance of gene transfer technology. Journal of Agricultural and Resource Economics, 29(3), 567–583.
Peters, E., Slovic, P., Hibbard, J. H., & Tusler, M. (2006). Why worry? Worry, risk perceptions, and willingness to act to reduce medical errors. Health Psychology, 25(2), 144–152.
Piazza, J., Landy, J. F., Chakroff, A., Young, L., & Wasserman, E. (2018). What disgust does and does not do for moral cognition. In N. Strohminger & V. Kumar (Eds.), The Moral Psychology of Disgust (pp. 53–81). New York: Rowman & Littlefield.
Prati, G., Pietrantoni, L., & Zani, B. (2012). Influenza vaccination: The persuasiveness of messages among people aged 65 years and older. Health Communication, 27(5), 413- 420.
Richardson, J. T. (2011). The analysis of 2 X 2 contingency tables - yet again. Statistics in Medicine, 30(8), 890.
Royzman, E. B., Atanasov, P., Landy, J. F., Parks, A., & Gepty, A. (2014). CAD or MAD? Anger (not disgust) as the predominant response to pathogen-free violations of the divinity code. Emotion, 14(5), 892–907.
Royzman, E. B., Cusimano, C., & Leeman, R. F. (2017). What lies beneath? Fear vs. disgust as affective predictors of absolutist opposition to genetically modified food and other new technologies. Judgment and Decision Making, 12(5), 466–480.
Royzman, E. B., Kim, K., & Leeman R. F. (2015). The curious tale of Julie and Mark: Unraveling the moral dumbfounding effect. Judgment and Decision Making, 10(4), 296- 313.
Royzman, E. B., Leeman, R. F., & Baron, J. (2009). Unsentimental ethics: Towards a content- specific account of the moral-conventional distinction. Cognition, 112(1), 159–174.
Royzman, E. B., Leeman, R. F., & Sabini, J. (2008). “You make me sick”: Moral dyspepsia as a reaction to third-party sibling incest. Motivation and Emotion, 32(2), 100–108.
Royzman, E. B., & Sabini, J. (2001). Something it takes to be an emotion: The interesting case of disgust. Journal for the Theory of Social Behavior, 31(1), 29–59.
Rozin, P., & Fallon, A. E. (1987). A perspective on disgust. Psychological Review, 94(1), 23–41.
Rozin, P., Haidt, J., & Fincher, K. (2009). From oral to moral: Is moral disgust an elaboration of a food rejection system. Science, 323(5918), 1179–1180
Rozin, P., Haidt, J., & McCauley, C. R. (1993). Disgust. In M. Lewis & J. Haviland (Eds.). Handbook of emotions (pp. 575–594). New York: Guilford.
Rozin, P., Haidt, J., & McCauley, C. R. (2000). Disgust. Entry in: D. Levinson, J. Ponzetti, & P. Jorgenson (eds.) Encyclopedia of human emotions. Volume 1 (second edition) (pp. 188–193). New York: Macmillan.
Rozin, P., Lowery, L., & Ebert, R. (1994). Varieties of disgust faces and the structure of disgust. Journal of Personality and Social Psychology, 66(5),870–888.
Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of cross-cultural studies. Psychological Bulletin, 115, 102–141.
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172.
Rzymski, P., & Królczyk, A. (2016). Attitudes toward genetically modified organisms in poland: to gmo or not to gmo? Food Security, 8(3), 689–697.
Scott, S. E., Inbar, Y., & Rozin, P. (2016). Evidence for moral opposition to genetically modified food in the united states. Perspectives on Psychological Science, 11(3), 315–324.
Seligman, M. E.P., Railton, P., Baumeister, R. F., & Sripada, C. (2013). Navigating into the future or driven by the past. Perspectives on Psychological Science, 8(2), 119–141.
Sherman, G., Haidt, J., & Coan, J. ( (2007). Nazis really are disgusting: Psychophysiological evidence for sociomoral disgust. Unpublished manuscript. University of Virginia.
Slovic, P. (Ed.). (2000). Risk, society, and policy series. The perception of risk. London, England: Earthscan Publications.
Tybur, J. M., Lieberman, D., Griskevicius, V. (2009). Microbes, mating, and morality: individual differences in three functional domains of disgust. Journal of Personality and Social Psychology, 97(1), 103–122.
van Huis, A. (2017). Did early humans consume insects? Journal of Insects as Food and Feed, 3(3), 161–163.
Watt, M. C., Maitland, R. A., & Gallagher, C. E. (2017). A case of the “heeby jeebies”: An examination of intuitive judgements of “creepiness”. Canadian Journal of Behavioural Science / Revue canadienne des sciences du comportement, 49(1), 58–69.
Widen, S. C., Pochedly, J. T., Pieloch, K., Russell, J. A. (2013). Introducing the sick face. Motivation and Emotion, 37(3) 550–557.
Woody, S. R., & Teachman, B. A. (2000). Intersection of disgust and fear: Normative and pathological views. Clinical Psychology: Science and Practice, 7(3), 291–311.
Copyright: © 2018. The authors license this article under the terms of the Creative Commons Attribution 3.0 License.
This document was translated from LATEX by HEVEA.