Judgment and Decision Making, vol. 8, no. 1, January 2013, pp. 29-33

Communicating clinical trial outcomes: Effects of presentation method on physicians’ evaluations of new treatments

Francesco Marcatto*   Jonathan J. Rolison#   Donatella Ferrante%

Physicians expect a treatment to be more effective when its clinical outcomes are described as relative rather than as absolute risk reductions. We examined whether effects of presentation method (relative vs. absolute risk reduction) remain when physicians are provided the baseline risk information, a vital piece of statistical information omitted in previous studies. Using a between-subjects design, ninety five physicians were presented the risk reduction associated with a fictitious treatment for hypertension either as an absolute risk reduction or as a relative risk reduction, with or without including baseline risk information. Physicians reported that the treatment would be more effective and that they would be more willing to prescribe it when its risk reduction was presented to them in relative rather than in absolute terms. The relative risk reduction was perceived as more effective than absolute risk reduction even when the baseline risk information was explicitly reported. We recommend that information about absolute risk reduction be made available to physicians in the reporting of clinical outcomes. Moreover, health professionals should be cognizant of the potential biasing effects of risk information presented in relative risk terms.


Keywords: relative risk, absolute risk, baseline risk, risk communication, presentation format.

1  Introduction

Most, if not all patients, consult with their physician regarding medical treatment, and the physician plays a major role in the treatment decisions of patients (Auerbach, 2001; Degner & Sloan, 1992). An important finding of recent years is that patients and the general public poorly interpret risk reductions associated with treatment options (see Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz & Woloshin, 2007, for a review). With patients relying on the recommendations of their physicians, we must ask whether physicians understand the reports of risk reduction behind the treatment options they recommend.

Risk reductions associated with medical treatment are typically reported either in relative or absolute terms. Relative risk reduction presents risks associated with a treatment relative to some baseline (e.g., a control group). A risk reduction from 10% to 5% for individuals administered a treatment represents a relative risk reduction of 50%, compared to an overall reduction of 5% in absolute terms. Perhaps not surprisingly, patients and people in general with no formal training in statistics anticipate treatments to be more effective when risk reductions are presented to them in relative than in absolute terms (Akl et al., 2011; Chao et al., 2003; Covey, 2007; Feinstein, 1992; Rolison, Hanoch & Miron-Shatz, 2012; Wegwarth, Gaissmaier & Gigerenzer, 2010). Of perhaps greater concern, physicians—who typically receive instruction in statistics as part of their basic medical training1—also succumb to the lure of relative risk reports, and are found to recommend treatment options more on basis of relative than of absolute risk reduction (Akl et al., 2011; Bobbio, Demichelis & Giustetto, 1994; Forrow, Taylor & Arnold, 1992; Moxey, O’Connell, McGettigan & Henry, 2003; Naylor, Chen & Strauss, 1992; Nexøe, Gyrd-Hansen, Kragstrup, Kristiansen & Nielsen, 2002).

Since physicians, patients, and the general public alike are unduly influenced by relative risk information, it is surprising that relative risk reduction is the most commonly used method of delivering statements about the efficacy of clinical trials in media reports (Moynihan et al., 2000), and in the technical reports on which physicians base their recommendations (Bucher, Weinbacher & Gyr, 1994; Fahey, Griffiths & Peters, 1995). In a survey of media reports, Moynihan et al. (2000) found that 83% of reports described risk reductions only in relative terms, compared to just 17% that provided the absolute risks.

Some authors suggest either discouraging the use of relative risk information, or combining it with absolute risk information (Laupacis, Sackett & Roberts, 1988; Rolison et al., 2012; see also Schwartz & Meslin, 2008), while others advocate policies for improving health literacy in the general public (Schwartz, Woloshin, Black & Welch, 1997). Despite the merits of these approaches, there still remains the possibility that physicians’ misunderstanding of relative risk reduction has been overestimated due to the default assumption, common to the previous studies, that physicians are aware of the baseline risks. Without the baseline risk, the efficacy of a treatment cannot be assessed, and any recommendations based on it may be misled. Depending on whether a baseline risk is 10%, 5%, or 1%, a relative risk reduction of 50% could mean that 50, 25, or 5 lives in every 1,000 will be saved by a treatment. Can we assume that physicians always know the baseline risks, especially when the baseline risks refer to domains outside the physician’s everyday practice? And when physicians know the baseline risk, is their understanding of the relative risk reduction accurate? Literature has showed that patients and the general public overestimate the relative risk reduction even when the baseline risk is explicitly provided (Chao et al., 2003; Wolf & Schorling, 2000). To date, however, no previous study has examined the influence of baseline risk information on physicians’ interpretations of relative risk information.

In the current article, we present physicians with reports of a fictitious clinical trial of a new treatment for hypertension—a chronic but treatable cardiovascular disease (Ong, Cheung, Man, Lau & Man, 2007). The report provides the risk reduction of the treatment either in terms of its absolute risk reduction, or in terms of its relative risk reduction with or without including the baseline risk information. Our aims were (a) to examine whether physicians’ willingness to prescribe a treatment and their ratings of its effectiveness would be influenced by the method in which risk reduction is provided (relative or absolute risk reduction), and (b) whether explicitly reporting the baseline risk in combination with information about relative risk reduction would influence the physicians’ willingness to prescribe and ratings of effectiveness.

2  Method


Figure 1: Physicians’ ratings of the effectiveness of a new treatment (Panel A) and their willingness to prescribe it (Panel B) by method of risk presentation. Bars represent 1 standard error below and above mean ratings. ARR = Absolute Risk Reduction, RRR = Relative Risk Reduction.
 

2.1  Participants

A total of 95 physicians (55 men, 40 women; mean age=37.27, SD=10.64), of which 41 were interns and 54 resident physicians at a public hospital in Northeast Italy were contacted at their workplace. All physicians agreed to participate in the study on a voluntary basis.

2.2  Materials and Procedure

Physicians were presented with one of three vignettes, adapted from a previous study by Forrow et al. (1992). The vignette reported the results of a clinical trial of a new drug treatment for hypertension. Physicians were randomly assigned to receive the vignette reporting the clinical trial outcome either as an absolute risk reduction, a relative risk reduction, or a relative risk reduction that included the baseline risk. The vignette that referred to an absolute risk reduction followed (translated from Italian; N=32):

A randomized controlled study of over 6,000 men with a moderate form of hypertension (DBP 90-104) conducted to evaluate the effectiveness of an experimental drug treatment, showed that the experimental drug treatment reduced the overall mortality rate over the 5 year period from 7.8% in the “routine treatment” control group to 6.3%. The 1.5% reduction in total mortality observed over the 5 year period was statistically significant.

The vignette that referred to a relative risk reduction followed (N=30):

A randomized controlled study of over 6,000 men with a moderate form of hypertension (DBP 90-104) conducted to evaluate the effectiveness of an experimental drug treatment, showed that the group treated with the experimental drug registered a reduction in the overall mortality rate over the 5 year period by 20.3% as compared with rate in the “routine treatment” control group. This reduction was statistically significant.

The vignette that referred to a relative risk reduction including the baseline risk followed (N=33):

A randomized controlled study of over 6,000 men with a moderate form of hypertension (DBP 90-104) conducted to evaluate the effectiveness of an experimental drug treatment, showed that the group treated with the experimental drug registered a reduction in the overall mortality rate over the 5 year period by 20.3% as compared with rate in the “routine treatment” control group (the “routine treatment” control group’s overall mortality rate over the 5 year period was equal to 7.8%). This reduction was statistically significant.

Regarding the vignette, physicians were asked to rate the effectiveness of the drug treatment on a 7 point scale ranging from −3 (“Not effective at all”) to 3 (“Very effective”), and their willingness to prescribe the drug treatment on the same 7 point scale from −3 (“Certainly not inclined to prescribe”) to 3 (“Certainly inclined to prescribe”).

3  Results

Physicians’ mean group ratings of treatment effectiveness and their willingness to prescribe are provided in Figure 1. A one way independent analysis of variance (ANOVA), including presentation method (absolute risk reduction, relative risk reduction, relative risk reduction including baseline risk) as an independent factor, and including physicians’ experience (residency vs. internship) and gender as covariates, was conducted separately on physicians’ ratings of effectiveness and their willingness to prescribe. The analysis revealed significant effects of presentation method both on physicians’ ratings of effectiveness (without covariates: F(2, 92)=13.53, MSe=0.71, p<.001, η2=.23; with covariates: F(2, 90)=13.02, MSe=0.71, p<.001, η2=.22) and their willingness to prescribe (without covariates: F(2, 90)=9.56, MSe=0.66, p<.001, η2=.17; with covariates: F(2, 90)=9.17, MSe=0.65, p<.001, η2=.16). There were no significant effects of gender (ratings of effectiveness, F(1, 90)=0.08, MSe=0.71, p=.773; willingness to prescribe, F(1, 90)=0.53, MSe=0.65, p=.470) or experience (ratings of effectiveness, F(1, 90)=1.78, MSe=0.71, p=.185; willingness to prescribe, F(1, 90)=2.86, MSe=0.71, p=.094).

Follow-up t-test comparisons (with Bonferroni correction) revealed significant mean group differences on ratings of effectiveness between the absolute risk reduction method and the relative risk reduction method, regardless of whether (p<.001) or not (p<.001) the baseline risk was included, such that physicians rated the treatment as more effective when information was provided as a relative (including baseline risk, M=2.24, SD=0.75; not including baseline risk, M=2.07, SD=0.64) rather than as an absolute risk reduction (M=1.22; SD=1.10). Regarding physicians’ willingness to prescribe, compared to the absolute risk reduction method (M=1.19, SD=0.90), physicians were significantly more willing to prescribe the treatment when presented as a relative risk reduction (including baseline risk, M=2.06, SD=0.70, p<.001; not including baseline risk, M=1.73, SD=0.83, p=.029). Including the baseline risk in combination with the relative risk reduction did not affect physicians’ ratings of effectiveness (p=.213) nor their willingness to prescribe (p=.205) compared with when the baseline risk was not included with the relative risk reduction.2

4  Discussion

The physician plays an important role in the treatment decisions of patients (Auerbach, 2001; Degner & Sloan, 1992). It is thus crucial that we examine how physicians evaluate reports of new clinical trial outcomes, and how their evaluations influence their willingness to prescribe new treatments to patients and their expectations about the effectiveness of new treatments.

In this study, we provided physicians the risk reduction associated with a new (but fictitious) treatment. Physicians were presented the treatment’s benefits as an absolute risk reduction, as a relative risk reduction, or as a relative risk reduction including the baseline risk. In this respect our study is unique. In fact previous studies (e.g., Forrow et al., 1992), as well as media reports of clinical outcomes (Moynihan et al., 2000) typically do not report the baseline risk, a crucial aspect of relative risk information. If the baseline risk is unknown, the relative risk reduction cannot be interpreted. We reasoned that participants of previous studies could have overestimated the effectiveness of treatments because they did not know the baseline risk information.

As expected, physicians claimed that the new treatment would be more effective and that they would be more willing to prescribe it when its risk reduction was presented to them in relative rather than in absolute terms. To our surprise, however, physicians appeared entirely unaffected by the addition of the baseline risk in combination with the relative risk reduction.

Why do physicians’ not take into account baseline risk information when evaluating clinical outcomes? We do not know whether physicians are not aware that the baseline risk information is relevant, or whether they do not have the statistical or numerical skills to integrate baseline risk information with relative risk reduction (see, for example, Barbey & Sloman, 2007; Kahneman & Tversky, 1973). It is clear, however, that physicians, like others, are unduly influenced by the “big” numbers that relative risk reductions present. Indeed, this may also explain the popular use of relative risk information in media reports of clinical outcomes (Moynihan et al., 2000).

It would seem that, even when all relevant statistical information is made available, the format in which risks are reported to physicians influences strongly their expectations of a treatment’s effectiveness and their willingness to prescribe a treatment to patients. Our current results echo the strong recommendations made by others that information about absolute risk reduction should be made available to physicians (as well as to the lay public) when information about relative risk reduction is used (Laupacis et al., 1988; Rolison et al., 2012). The lure of relative risk information may be especially strong among the general public who typically do not have direct training in interpreting health risk statistics (Galesic & Garcia-Retamero, 2010; Schwartz et al., 1997). Although better than the general public, physicians also show difficulties with numerical skills necessary for interpreting health risk statistics (Estrada, Barnes, Collins & Byrd, 1999). Thus, we suggest also that more extensive statistical and numerical training be made available to physicians.

In conclusion, based on our current results we strongly recommend that health professionals be cognizant of the potential biasing effects of risk information presented in relative risk terms. We urge that absolute risk information also be made available when clinical outcomes are reported to physicians.

References

Akl, E. A., Oxman, A. D., Herrin, J., Vist, G. E., Terrenato, I., Sperati, F., Costiniuk, C., Blank, D., & Schünemann, H. (2011) Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database of Systematic Reviews, Issue 3, Art. No.: CD006776.

Auerbach, S. M. (2001). Do patinets want control over their own health care? A review of measures, findings, and research issues. Journal of Health Psychology, 6, 191–203.

Barbey, A. K., & Sloman, S. A. (2007). Base-rate respect: From ecological rationality to dual processes. Behavioral and Brain Sciences, 30, 241–254.

Bobbio, M., Demichelis, B., & Giustetto, G. (1994). Completeness of reporting trial results: effect on physicians’ willingness to prescribe. The Lancet, 8907, 1209–1211.

Bucher, H. C., Weinbacher, M., & Gyr, K. (1994). Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration. British Medical Journal, 309, 761–764.

Chao, C., Studts, J. L., Abell, T., Hadley, T., Roetzer, L., Dineen, S., et al. (2003). Adjuvant chemotherapy for breast cancer: How presentation of recurrence risk influences decision-making. Journal of Clinical Oncology, 21, 4299-4305.

Covey, J. (2007). A meta-analysis of the effects of presenting treatment benefits in different formats. Medical Decision Making, 27, 638–654.

Degner, L. F., & Sloan, J. A. (1992). Decision making during serious illness: What role do patients really want to play? Journal of Clinical Epidemiology, 45, 941–950.

Estrada, C., Barnes, V., Collins, C., & Byrd, J.C. (1999). Health literacy and numeracy. Journal of the American Medical Association, 282, 527.

Fahey, T., Griffiths, S., & Peters, T. J. (1995). Evidence based purchasing: Understanding results of clinical trials and systematic reviews. British Medical Journal, 311, 1056–1059.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.

Feinstein, A. R. (1992). Invidious comparisons and unmet clinical challenges. The American Journal of Medicine, 92, 117–120.

Forrow, L., Taylor, W. C., & Arnold, R. M. (1992). Absolutely relative: How research results are summarized can affect treatment decisions. The American Journal of Medicine, 92, 121–124.

Galesic, M., Garcia-Retamero, R. (2010). Statistical numeracy for health: A cross-culural comparison with probabilistic national samples. Archives of Internal Medicine, 170, 462–468.

Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients to make sense of health statistics. Psychological Science in the Public Interest, 8, 53–96.

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237–251.

Laupacis, A., Sackett, D. L., & Roberts, R. (1988). An assessment of clinically useful measures of the consequences of treatment. New England Jounral of Medicine, 318, 1728–1733.

Moynihan, R., Bero, L., Ross-Degnan, D., Henry, D., Lee, K., Watkins, J., et al. (2000). Coverage by the news media of the benefits and risks of medications. New England Journal of Medicine, 342, 1645–1650.

Moxey, A., O’Connell, D., McGettigan, P., & Henry, D. (2003). Describing treatment effects to patients. Journal of General Internal Medicine, 18, 948–959.

Naylor, C. D., Chen, E., & Strauss, B. (1992). Measured Enthusiasm: Does the Method of Reporting Trial Results Alter Perceptions of Therapeutic Effectiveness? Annals of Internal Medicine, 117, 916–21.

Nexøe, J., Gyrd-Hansen. D., Kragstrup, J., Kristiansen, I. S., & Nielsen, J. B. (2002). Danish GPs’ perception of disease risk and benefit of prevention. Family Practice, 19, 3–6.

Ong, K. L., Cheung, B. M. Y., Man, Y. B., Lau, C. P., & Lam, K. S. L. (2007). Prevalence, awareness, treatment, and control of hypertension among United States adults 1999–2004. Hypertension, 49, 69–75.

Rolison, J. J., Hanoch, Y., & Miron-Shatz, T. (2012). What do men understand about lifetime risk following genetic testing? The effect of context and numeracy. Health Psychology, 31, 530–533.

Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine, 127, 966–972.

Schwartz, P. H., & Meslin, E. M. (2008). The Ethics of Information: Absolute Risk Reduction and Patient Understanding of Screening. Journal of General Internal Medicine, 23, 867–70.

Wegwarth, O., Gaissmaier, W., & Gigerenzer, G. (2010). Deceiving numbers: Survival rates and their impact on doctors’ risk communication. Medical Decision Making, 31, 386–394.

Wolf, A. M. D., & Schorling, J. B. (2000). Does Informed Consent Alter Elderly Patients’ Preferences for Colorectal Cancer Screening? Journal of General Internal Medicine, 15, 24–30.

World Federation for Medical Education. (2012) Basic Medical Education. WFME Global Standards for Quality Improvement. The 2012 Revision. Retrieved December 11, 2012, from the World Federation for Medical Education Web site: http://www.wfme.org/standards/bme


*
Department of Life Sciences, University of Trieste, Via Weiss 21, 34138 Trieste, Italy. Email: fmarcatto@units.it.
#
Department of Psychology, Queen’s University Belfast, UK.
%
Department of Life Sciences, University of Trieste, Italy.
We thank Fabio Del Missier and Yaniv Hanoch for helpful comments on an earlier version of this article and Manuela Slavec for her assistance with data collection.
1
According to the WFME Global Standards for Quality Improvement, basic medical education curricula must incorporate behavioral and social sciences, including biostatistics and epidemiology, with the aim to enable effective communication and clinical decision making (World Federation for Medical Education, 2012).
2
A power analysis using G*Power 3 software (Faul, Erdfelder, Lang & Buchner, 2007) revealed that, in order to detect a medium effect size (d = .50) between two groups, with α =.05 and power =.95, each group required at least 27 participants. Thus, the sample sizes used in this study should have been adequate to detect a medium sized effect.

This document was translated from LATEX by HEVEA.