Judgment and Decision Making, Vol. 17, No. 2, March 2022, pp. 315-330
Does the evaluability bias hold when giving to animal charities?Glen William Spiteri* # $ |
Abstract:When evaluating a charity by itself, people tend to overweight overhead costs in relation to cost-effectiveness. However, when evaluating charities side by side, they base their donations on cost-effectiveness. I conducted a replication and extension of Caviola et al. (2014; Study 1) using a 3 (High Overhead/Effectiveness, Low Overhead/Effectiveness, Both) x 2 (Humans, Animals) between-subjects design. I found that the overhead ratio is an easier attribute to evaluate than cost-effectiveness in separate evaluation, and, in joint evaluation, people allocate donations based on cost-effectiveness. This effect was observed for human charities, and to a lesser extent, for animal charities.
Keywords: animal charity, evaluability bias, overhead aversion, cost-effectiveness, preference reversal, pre-registration, altruism.
The study behind donors’ reluctance to cover running costs with their donation money – known as overhead aversion – is well-documented in the literature, (Bowman, 2006; Meer, 2014; Caviola et al., 2014; Gneezy et al., 2014; Portillo & Stinn, 2018; Behavioural Insights Team, 2019; Baron & Szymanska, 2011; MacAskill, 2016; Caviola et al., 2021). However, this narrative might fuel the perception of overheads as “wasteful spending”. Ultimately, charities need to spend on overheads to hire skilled staff and have good infrastructure to fulfil their mission (Caviola et al., 2020). Thus, to drive effective giving, donors should direct their attention towards cost-effectiveness rather than levels of overhead (Singer, 2015; Caviola et al., 2020).
Nevertheless, donors often base their allocation decisions on easy-to-evaluate information about operating costs (Bazerman et al., 1992; Baron & Szymanska, 2011). Building on Hsee’s (1996) evaluability hypothesis, Caviola et al. (2014) call this phenomenon the evaluability bias and define it as “the tendency to weight the importance of an attribute in proportion to its ease of evaluation, rather than based on criteria that are deemed as more relevant after reflection” (p. 304). In the context of altruistic decision-making, Caviola et al. (2014) did four studies to test for evidence of the evaluability bias. In one of them, they asked their recruited participants from MTurk how much they would be willing to donate to two hypothetical charities that saved humans. The charities were presented with information on administration expenses and cost-effectiveness. Participants were shown either Charity A with high overhead and high effectiveness (separate evaluation), Charity B with low overhead and low effectiveness (separate evaluation), or both (joint-evaluation). When presented in isolation, donations were higher to Charity B. In contrast, joint-evaluation contributions were higher to the more effective Charity A. Thus, the authors provide evidence that when people are given a reference point (presenting both charities side by side), they become more concerned with effectiveness and hence become de-biased.
The present article replicates Study 1 by Caviola et al. (2014), and extends the original design by including charities that save animals. This extension comes as a result of the perception that animals suffer less, have lower cognitive capacity than humans, and are valued less than humans (Caviola et al., 2020, 2019; Liebe & Jahnke, 2017; Johansson-Stenman, 2018). Thus, are donors less sensitive to overhead in the case of animals?
I expected that, even in the case of animal charities, the overhead ratio is an easier attribute to evaluate than cost-effectiveness. When evaluated jointly, cost-effectiveness is an easier attribute to evaluate than the overhead ratio.
250 UK subjects with a mean age of 32 years (SD=11.68) were recruited through Prolific (Palan & Schitter, 2018; Chandler et al., 2015) against a remuneration of £7.50/hr in return for their participation. In the pre-registration stage, I planned to exclude participants based on two attention checks and incompletion.1
I used a between-subjects design similar to Hsee (1996), Erlandsson (2021), and Caviola et al. (2014) over an online survey in Qualtrics (see Appendix ??). Upon entry, subjects were first presented with an information sheet and asked for consent. Following this, subjects were randomly allocated into one of two studies: Humans or Animals, and one of three treatments in equal numbers (see Figure ??).2 The charities across both studies were perfectly identical, the only difference being that those in the Humans study dealt with saving human lives, whereas those in Animals saved animal lives.3 Participants’ donation decisions were hypothetical, but part of their amounts would be transferred to real charities.
In accordance with ethical guidelines, participants were presented with information text on the study and asked for their consent. If subjects did not consent to participation, then they were redirected to Prolific; if they consented, they were taken to the instruction block. Here, participants were reminded that their hypothetical donation amounts would translate into real-world impact as 5% of all responses were donated to the Malaria Consortium and Animal Charity Evaluators.4 The instructions included the following statements: “You will now be asked to think about a hypothetical scenario in the charitable giving domain. You will be asked to allocate donation amounts. Your donation amounts will have a real-world impact as a portion of your donations will be donated to charity.” Next, participants were asked to write that they have read the instructions, acting as a prime (Dolan et al., 2012), and serving as the first attention check of the study. Subsequently, subjects were taken to the main experimental manipulation which followed that of Caviola et al. (2014) and Hsee (1996). 125 UK subjects were assigned to Humans with a mean age of 32 years (SD=12.13). Of these, 2 failed at least one attention check. A further 125 UK subjects were assigned to Animals with a mean age of 33 years (SD=11.16). One participant was excluded because they failed one attention check.
Subjects were randomly assigned to an experimental group and prompted to allocate an amount between £0 and £100 to each charity (similar to Caviola et al., 2019). In the first two treatment groups High and Low, participants were shown either one of the two charities in isolation, with their respective attributes: cost-effectiveness and administrative expenses (see Caviola et al., 2014).5 The questions were presented as follows:
High overhead/High effectiveness “Out of a donation of £100, a charity spends £60 on administrative expenses. With the remaining £40, 5 human [animal] lives are saved. In other words, the charity saves 5 human [animal] lives with an overhead ratio of 60%. How much would you be willing to donate to this charity?”
Low overhead/Low effectiveness “Out of a donation of £100, a charity spends £5 on administrative expenses. With the remaining £95, 2 human [animal] lives are saved. In other words, the charity saves 2 human [animal] lives with an overhead ratio of 5%. How much would you be willing to donate to this charity?”
Joint Evaluation “Charity A: Out of a donation of £100, the charity spends £60 on administrative expenses. With the remaining £40, 5 human [animal] lives are saved. In other words, the charity saves 5 human [animal] lives with an overhead ratio of 60%. Charity B: Out of a donation of £100, the charity spends £5 on administrative expenses. With the remaining £95, 2 human [animal] lives are saved. In other words, the charity saves 2 human [animal] lives with an overhead ratio of 5%. How much would you be willing to donate to each charity?"
After being presented with information about administrative expenses, effectiveness, and overhead ratios, subjects were asked to indicate their hypothetical donation amounts restricted with an endowment of £100.6 Similar studies on the evaluability hypothesis (such as Caviola et al., 2014; Hsee, 1996) used ranges from $10 to $50 or $100 to $500, implying that my range (£0 to £100) is broadly comparable.7 Information presented was adjusted accordingly for consistency. Finally, participants were presented with a seriousness check (Aust et al., 2013), and general demographic questions to gauge sample representativeness.
Concerning the question of whether donors to human charities tend to weigh the importance of an attribute in proportion to its ease of evaluation, Figure 2 shows their average donations. Donations in the separate evaluation condition are shown in the left-hand panel. When evaluated separately, independent sample t-tests revealed that subjects donated significantly more to the Low overhead/effectiveness charity (M=78.6, SD=32.5) than they did to the High overhead/effectiveness charity (M=64.6, SD=41.4; t(83)=–1.72, p=.045). To test whether this held when both charities were presented jointly, I report donations in joint-evaluation on the right-hand panel in Figure ??. In this case, subjects donated significantly more to the High overhead/effectiveness charity (M=65.1, SD=30.9) than they did to the Low overhead/effectiveness charity (M=34.9, SD=30.9; paird t-test [t(39)=3.10, p=.002). Thus, donors showed a preference for the Low overhead/effectiveness charity when presented in isolation, whereas when evaluated jointly, donors donated more to the High overhead/effectiveness charity. This observation supports Hsee’s (1996) preference reversal (t(120)=4.53, p=.001), i.e., preferences change when given a reference point (Hsee, 1998; Hsee et al., 1999).8. Furthermore, the results lend support for the hypothesis that donors find overhead to be an easier attribute to evaluate than effectiveness, confirming the existence of the evaluability bias observed by Caviola et al. (2014).
In Animals, I attempt to answer the same question as in Humans to probe the question of whether the evaluability bias holds in the context of animal charities. Figure 3 shows average donation amounts across separate evaluation and joint-evaluation conditions. A priori, I suspected that donors in the Animals study would be equally sensitive to overhead ratios as donors in the Humans study. In fact, I find support for the hypothesis that donors to animal charities find the overhead ratio to be more evaluable than effectiveness, albeit at a weaker level of significance. In separate evaluation, an independent sample t-test revealed that donors donated more to the Low overhead/effectiveness charity (M=61.6, SD=41.7) than they did to the High overhead/effectiveness (M=48.0, SD=36.9; t(83)=–1.4, p=.078 two tailed). Once again, donors exhibited unstable preferences when the evaluation mode was changed, demonstrating preference reversal (t(119)=3.04, p=.003). A paired sample t-test for the joint-evaluation condition revealed that donors donated more to the High overhead/effectiveness charity (M=58.7, SD=34.7) than to the Low overhead/effectiveness charity (M=41.3, SD=34.7; t(40)=1.61, p=.058).
The present article carried out a replication of one of the four studies by Caviola et al. (2014). It tested for evidence of the evaluability bias when giving to human charities, and extended it by investigating whether the same effect can be applied to animal charitable giving. This was carried out by using the same scenario as in Caviola et al. (2014), and eliciting hypothetical donation amounts from participants. By way of recap, analysis of means finds that (i) the overhead ratio is an easier attribute to evaluate than cost-effectiveness in human charities, and (ii) when evaluated jointly, cost-effectiveness is more accessible than the overhead ratio. In the case of animal charities, the same effect was observed but at marginal significance.
In the first study dealing with saving human life, the results indicate that contrary to much of the literature, donors are not preoccupied with overhead per se (Caviola et al., 2014). Instead, donors find the overhead ratio easier to evaluate than cost-effectiveness when they do not have a reference point. When judging their options in isolation, donors donated more money to the low overhead charity, but donors’ preferences reversed when the two charities were presented jointly (Hsee, 1996; Caviola et al., 2014; Erlandsson, 2021; Hsee, 2010). In joint-evaluation, donors donated more money to the more cost-effective charity, despite the higher overhead. The most likely explanation behind this is that donors are given a reference point to compare with and are effectively de-biased against overhead (Berman et al., 2018). In the second study, the picture is less clear. When dealing with animal lives, results indicate weak evidence in favour of the evaluability bias. Similar to the first study, donors donated more money to the charity with low overhead when evaluated in isolation. Then, when evaluated jointly, donors’ preferences shifted towards the more effective charity. However, the results were only marginally significant. The likely reason behind the weaker effect is that cost-effectiveness information was given in terms of animal lives saved in general. In other words, the experiment employed a generic animal category, therefore donors’ attitudes towards animal life might have varied based on their imagined animal category.
In such experimental methods, a fundamental shortcoming is the issue of external validity. There may be little connection between responses here and those in real life. Subjects’ responses were hypothetical and might differ from real world behaviour. To partly mitigate this, participants were told that their donations had a real impact as I donated a portion of their responses to real charities.
There are also issues endemic to my experiments. First, the presentation of cost-effectiveness and overhead likely increased the salience and relative evaluability of both attributes. In the field, donors are presented with particular characteristics and a vast array of stimuli that might drive their donations (Berman et al., 2018). A second issue is that the Animals study did not refer to a particular kind of animal species. In the experiment, a generic animal category was used, possibly accounting for the diminished evaluability bias. A third issue is that despite being sufficiently-powered, the sample sizes are quite small. Fourth, when evaluating charities jointly, participants could have been motivated by fairness concerns in their donation allocations (Li & Hsee, 2019; Erlandsson et al. 2020; Hsee et al., 2005).
There is merit in extending the research agenda to explore whether the evaluability bias continues to hold in different contexts for different causes. More practically, research has thus far operated in a hypothetical setting and possibly real world behaviour might differ, especially when donors have more than two options to choose from. In light of this, future research would benefit from randomised controlled trials (RCTs) in the field where donors are observed in a real-world environment. Secondly, there is a demonstrable relationship between levels of overhead, effectiveness and donations. However, research is less clear about the extent to which donors would be willing to trade-off overhead for effectiveness (Mott et al., 2020; Johansson-Stenman, 2018; Benjamin et al., 2014). Third, future research can look at the animals question using specific animal species, and employing larger sample sizes.
In this study, I empirically investigated the evaluability bias in charitable animal giving. Based on previous research by Caviola et al. (2014), I hypothesised that the evaluability bias would be replicated in a UK sample recruited via Prolific. Secondly, I extended this claim by hypothesising that the evaluability bias will be as strong for animal charities as for human charities. Results showed that the evaluability bias indeed affects decisions about human charitable giving. However, in the case of charitable animal giving, the effect was marginal.
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45(2), 527–535.
Baron, J., & Szymanska, E. (2011). Heuristics and biases in charity, In D. M. Oppenheiemer & C. Y. Olivola (Eds.), The science of giving: Experimental approaches to the study of charity, pp. 215–235. New York, NY: Psychology Press; US.
Bartels, D. M., & Burnett, R. C. (2011). A group construal account of drop-in-the-bucket thinking in policy preference and moral judgment. Journal of Experimental Social Psychology, 47(1), 50-57.
Bazerman, M. H., Loewenstein, G. F., & White, S. B. (1992). Reversals of preference in allocation decisions: Judging an alternative versus choosing among alternatives. Administrative Science Quarterly, 37(2), 220–240.
Behavioural Insights Team (2019). To the core: Charity futures. London: Cabinet Office. https://www.behaviouralinsights.co.uk/wp-content/uploads/2018/06/CHARITY_FUTURES_TOTHECORE_REPORT_v2-1.pdf
Benjamin, D. J., Heffetz, O., Kimball, M. S., & Rees-Jones, A. (2014). Can marginal rates of substitution be inferred from happiness data? Evidence from residency choices. The American Economic Review, 104(11), 3498–3528.
Berman, J. Z., Barasch, A., Levine, E. E., & Small, D. A. (2018). Impediments to effective altruism: the role of subjective preferences in charitable giving. Psychological Science, 29(5), 834–844.
Bowman, W. (2006). Should donors care about overhead costs? Do they care? Nonprofit and Voluntary Sector Quarterly, 35(2), 288–310.
Caviola, L., Faulmüller, N., Everett, J. A. C., Savulescu, J., & Kahane, G. (2014). The evaluability bias in charitable giving: Saving administration costs or saving lives? Judgment and Decision Making, 9,(4) 303–315.
Caviola, L., Everett, J. A. C., & Faber, N. S. (2019). The moral standing of animals: Towards a psychology of speciesism. Journal of Personality and Social Psychology, 116(6), 1011–1029.
Caviola, L., Everett, J. A. C., Kahane, G., Teperman, E., & Savulescu, J. (2020). The moral standing of animals: Towards a psychology of speciesism. Journal of Experimental Psychology: General, 1–32.
Caviola, L., Schubert, S., & Nemirow, J. (2020). The many obstacles to effective giving. Judgment and Decision Making, 15(2) 159–172.
Caviola, L., Schubert, S., Teperman, E., Moss, D., Greenberg, S., & Faber, N. S. (2020). Donors vastly underestimate differences in charities’ effectiveness. Judgment and Decision Making, 15(4) 509–516.
Caviola, L., Schubert, S., & Greene, J. D. (2021). The psychology of (in)effective altruism. Tends in Cognitive Sciences, 25(7), 596–607.
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7) 1131–1139.
Dolan, P., Hallsworth, M., Halpern, D., King, D., Metcalfe, R., & Vlaev, I. (2012). Influecing behaviour: The mindspace way. Journal of Economic Psychology, 33(1), 264–277.
Erlandsson, A., Lindkvist, A., Lundqvist, K., Andersson, P. A., Dickert, S., Slovic, P., & Västfjäll, D. (2020). Moral preferences in helping dilemmas expressed by matching and forced choice. Judgment and Decision Making, 15(4), 452–475.
Erlandsson, A. (2021). Seven (weak and strong) helping effects systematically tested in separate evaluation, joint evaluation and forced choice. Judgment and Decision Making, 16(5), 1113–1154.
Gneezy, U., Keenan, E. A., & Gneezy, A. (2014). Avoiding overhead aversion in charity. Science, 346(6209), 632–635.
Hsee, C. K. (1996). The evaluability hypothesis: An explanation for preference reversals between joint and separate evaluations of alternatives. Organizational Behavior and Human Decision Processes, 67(3), 247–257.
Hsee, C. K. (1998). Less is better: when low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11(2), 107–121.
Hsee, C. K., Loewenstein, G., Blount, S., & Bazerman, M. (1999). Preference reversals between joint and separate evaluations of options: A review and theoretical analysis. Psychological Bulletin, 125(5), 576–590.
Hsee, C. K., Rottenstreich, Y., & Xiao, Z. (2005). When is more better? On the relationship between magnitude and subjective value. Current Directions in Psychological Science, 14(5), 234–237.
Hsee, C. K., & Zhang, J. (2010). General evaluability theory. Perspectives on Psychological Science, 5(4), 345–355.
Johansson-Stenman, O. (2018). Animal welfare and social decisions: Is it time to take bentham seriously? Ecological Economics, 145, 90–103.
Liebe, U., & Jahnke, B. (2017). Giving more to humans than to animals in need? A behavioral measure of animal-human continuity in large-scale surveys. Anthrozoös, 30(2), 249–261.
Li, X., & Hsee, C. K. (2019). Beyond preference reversal: Distinguishing justifiability from evaluability in joint versus single evaluations. Organizational Behavior and Human Decision Processes, 153, 63–74.
MacAskill, W. (2016). Doing good better: How effective altruism can help you help others, do work that matters, and make smarter choices about giving back. Penguin.
Meer, J. (2012). Effects of the price of charitable giving: Evidence from an online crowdfunding platform. Journal of Economic Behavior & Organization, 103(1), 113–124.
Mott, D. J., Chami, N., & Tervonen, T. (2014). Reporting Quality of Marginal Rates of Substitution in Discrete Choice Experiments That Elicit Patient Preferences. Value in Health, 23(8), 979–984.
Palan, S., & Schitter, C. (2018). Prolific.ac — A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.
Portillo, J. E., & Stinn, J. (2018). Overhead aversion: Do some types of overhead matter more than others? Journal of Behavioral and Experimental Economics, 72, 40–50.
Singer, P. (2015). The Most Good You Can Do. How effective altruism is changing ideas about living ethically. Yale University Press.
The following is a transcription of the Qualtrics survey sent out to participants.
Dear Participant,
This study is about charitable giving.
Before you decide to participate, please read the following instructions to confirm that you agree to take part. This is a requirement by the University.
Your participation will consist of a simple electronic questionnaire and will take approximately 5 minutes of your time. This project has been approved by the Research Ethics Committee at the London School of Economics and Political Science. There are no risks associated with this study. If you have any questions, you may contact the Research Ethics Manager via research.ethics@lse.ac.uk.
Do not hesitate to ask any questions or forward your concerns about the study either before, during or after your participation by contacting me on [ ]. Please note that your responses are treated anonymously and will be used for research purposes only. You will be paid upon completion of the survey. If you wish to withdraw your consent at any point, you may do so without giving any reason and without any consequences.
Please indicate below whether you would like to participate in this study.
End of Block: Consent Block
Thank you for agreeing to participate in this study.
You will now be asked to think about a hypothetical scenario in the charitable giving domain. You will be asked to allocate donation amounts.
In the text box below, kindly write "I read the instructions":
End of Block: Preamble
Out of a donation of £100, a charity spends £60 on administrative expenses. With the remaining £40, 5 human lives are saved.
In other words, the charity saves 5 human lives with an overhead ratio of 60%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to this charity?
Out of a donation of £100, a charity spends £5 on administrative expenses. With the remaining £95, 2 human lives are saved.
In other words, the charity saves 2 human lives with an overhead ratio of 5%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to this charity?
Imagine that you want to donate to charity and you are presented with the following two charitable organisations.
Out of a donation of £100, a charity spends £60 on administrative expenses. With the remaining £40, 5 human lives are saved. In other words, the charity saves 5 human lives with an overhead ratio of 60%.
Out of a donation of £100, a charity spends £5 on administrative expenses. With the remaining £95, 2 human lives are saved. In other words, the charity saves 2 human lives with an overhead ratio of 5%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to each charity?
End of Block: Study 1 - Humans
Out of a donation of £100, a charity spends £60 on administrative expenses. With the remaining £40, 5 animal lives are saved.
In other words, the charity saves 5 animal lives with an overhead ratio of 60%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to this charity?
Out of a donation of £100, a charity spends £5 on administrative expenses. With the remaining £95, 2 animal lives are saved.
In other words, the charity saves 2 animal lives with an overhead ratio of 5%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to this charity?
Imagine that you want to donate to charity and you are presented with the following two charitable organisations.
Out of a donation of £100, a charity spends £60 on administrative expenses. With the remaining £40, 5 animal lives are saved. In other words, the charity saves 5 animal lives with an overhead ratio of 60%.
Out of a donation of £100, a charity spends £5 on administrative expenses. With the remaining £95, 2 animal lives are saved. In other words, the charity saves 2 animal lives with an overhead ratio of 5%.
You may choose to donate any amount between £0 and £100. Your donation amount cannot exceed £100.
How much would you be willing to donate to each charity?
End of Block: Study 2 - Animals
Below are a series of statements with which you may either agree or disagree. For each statement, please indicate the degree of your agreement/disagreement. Remember that your first responses are usually the most accurate.
[Answers were on a 7-point scale:
Strongly agree; Agree; Somewhat agree; Neither agree nor disagree;
Somewhat disagree; Disagree; Strongly disagree.]
Please click scale point 1 to confirm that you are paying attention.
Morally, animals always count for less than humans.
Humans have the right to use animals however they want to.
It is morally acceptable to keep animals in circuses for human entertainment.
It is morally acceptable to trade animals like possessions.
Chimpanzees should have basic legal rights such as a right to life or a prohibition of torture.
It is morally acceptable to perform medical experiments on animals that we would not perform on any human.
End of Block: Speciesism Scale
End of Block: Demographic Questions
Please answer the following questions about your political beliefs.
[Answers were on a 7-point scale:
Very Liberal; Liberal; Somewhat Liberal; Moderate; Somewhat Conservative; Conservative; Very Conservative.]
How economically liberal or conservative are you?
Please click scale point 1 to confirm that you are paying attention.
How socially liberal or conservative are you?
End of Block: Religiosity and Political Beliefs Scales
End of Block: Seriousness Check
This document was translated from LATEX by HEVEA.
