Judgment and Decision Making, Vol. 10, No. 4, July 2015, pp. 284-295
Improving dynamic decision making through training and self-reflectionSarah J. Donovan* C. Dominik Güss# Dag Naslund$ |
The modern business environment requires managers to make effective decisions in a dynamic and uncertain world. How can such dynamic decision making (DDM) improve? The current study investigated the effects of brief training aimed at improving DDM skills in a virtual DDM task. The training addressed the DDM process, stressed the importance of self-reflection in DDM, and provided 3 self-reflective questions to guide participants during the task. Additionally, we explored whether participants low or high in self-reflection would perform better in the task and whether participants low or high in self-reflection would benefit more from the training. The study also explored possible strategic differences between participants related to training and self-reflection. Participants were 68 graduate business students. They individually managed a computer-simulated chocolate production company called CHOCO FINE and answered surveys to assess self-reflection and demographics. Training in DDM led to better performance, including the ability to solve initial problems more successfully and to make appropriate adjustments to market changes. Participants’ self-reflection scores also predicted performance in this virtual business company. High self-reflection was also related to more consistency in planning and decision making. Participants low in self-reflection benefitted the most from training. Organizations could use DDM training to establish and promote a culture that values self-reflective decision making.
Keywords: dynamic decision making, complex problem solving, training, self-reflection, microworlds, strategies.
Many professions and situations require people to make time-pressured decisions for novel problems with vague or competing goals. An army unit commander, a juror, and a CEO are similar in that they all make highly consequential decisions under these circumstances. Dynamic decision-making (DDM) skills should help decision makers process information, formulate flexible action plans, and balance multiple objectives in many real world problems (BIBB, 2005). DDM can be defined as making a series of interdependent decisions in an environment that changes over time due to the consequences of the decisions made or due to autonomous changes in the environment (Brehmer, 1992; Fischer, Greiff & Funke, 2012; Gonzalez, Vanyukov & Martin, 2005). The goals of the current study are to demonstrate that self-reflection improves DDM performance and that brief training in DDM steps and self-reflection can also improve performance. Additionally, the study includes an in-depth analysis that explores how self-reflection and training could affect decision-making strategies.
The importance of understanding and improving DDM is evident in various research domains including economics, education, engineering, ergonomics, human-computer interaction, management, and psychology (Osman, 2010). Within psychology, DDM has been studied in the real world in the naturalistic decision-making (NDM) paradigm (e.g., Klein, 1998) and in computer simulated task environments or microworlds in the complex problem-solving (CPS) paradigm (e.g., Dörner, 1996; Frensch & Funke, 1995; Funke, 2003, 2010; Güss & Dörner, 2011). The practices within each of these two paradigms complement each other: NDM makes observations during field research and develops models, while CPS forms and tests hypotheses in the laboratory. Hypothesis testing generally uses the individual differences approach and tests for correlations between cognitive (e.g., intelligence) or personality variables (e.g., openness, extraversion) and performance in DDM tasks (Brehmer & Dörner, 1993; Güss, 2011; Schaub, 2001). An ongoing challenge for researchers is to uncover the underlying factors that differentiate performance in DDM tasks.
Adult decision makers have the cognitive ability to work through complex and dynamic problems, but often show cognitive biases and errors (Dörner, 1996; Ramnarayan et al., 1997). Research associates self-reflection with a reduction in these common biases and errors (Güss, Evans, Murray & Schaub, 2009; Locke & Latham, 2002; Osman, 2010). Self-reflection is “the evaluation of one’s thoughts, feelings, and behaviors” (Grant et al., 2002, p. 821).
Self-reflective decision making requires decision makers to consciously and continuously reflect on themselves and the situation (Locke & Latham, 2006; Sanders & McKeown, 2008). Self-reflection should help decision makers adapt to novel environments and situations because it facilitates their ability to relate new information to prior knowledge and to understand ideas and feelings (Sanders & McKeown, 2008; Campitelli & Labollita, 2010). Thus, self-reflection is a critical process for the reason that it enables the decision maker to make strategic adjustments to situational changes.
Self-reflection has often been understood as a trait. The evaluation of one’s own thoughts, feelings, and behaviors can be regarded as an individual difference variable—if self-monitoring can be seen as an indicator for self-reflection (e.g., Snyder, 1974). Self-reflection can, however, also be understood as a state. People may engage in self-reflection depending on the importance and relevance of a task. Self-reflection might be more dominant when buying a car compared to when buying cereal in a grocery store. Research on training programs on metacognition and critical thinking speak for the view of self-reflection as a state. Such research has shown that self-reflection can be modified (e.g., Ford et al., 1998; Helsdingen et al., 2010).
The ability and motivation of decision makers to use self-reflection varies among tasks as well as individuals (Güss et al., 2009; Sanders & McKeown, 2008). Güss et al. (2009) asked participants acting as firefighters in the microworld FIRE to answer three reflective questions and found that participants who received these aids performed better than those who did not receive aids or who worked on an unrelated task during a break. The three questions were: Which aspects of the game do I understand well? Which aspects of the game do I not understand well? When I go back to the game, what will I do differently to increase my performance? When Güss et al. (2010) analyzed DDM in two microworlds using think-aloud protocols and did not explicitly instruct participants to self-reflect, the researchers found that participants made few self-reflective statements.
Self-reflection can benefit each step of the DDM and problem-solving process. Researchers (Güss et al., 2009, Güss & Dörner, 2011; Klein, 1998; Sternberg, 1986) agree on the steps (although sometimes using different terminology): 1) problem identification and goal definition; 2) information gathering; 3) elaboration and prediction (forecasting); 4) strategic and tactical planning; 5) decision making and action; 6) evaluation of outcome with possible modification of strategy. The frequency and duration of each subsequent step depends on task characteristics and decision-maker preferences (Güss et al., 2010).
First, decision makers identify the problem and define adequate problem solving goals. Goals like “do your best” or “learn the system” can facilitate learning by reducing performance anxiety and enhancing self-regulatory behaviors (Locke & Latham, 2006; Osman, 2011). Through goal-focused self-reflection, decision makers should come to understand the strengths and weaknesses of their decision making and gain insight and control (Grant et al., 2002; Sanders & McKeown, 2008). Although the main goal may seem clear, i.e., make profit, subgoals need to be developed through self-reflection with regard to exactly how the main goal can be accomplished.
Decision makers in DDM tasks must gather situational information relevant to their goals in order to see if and how causal relationships change over time (Ramnarayan et al., 1997). Self-reflection should promote curiosity and exploration of contingencies within a task environment and prompt insight into the task at hand.
In elaboration and prediction, decision makers infer some aspects of the problem environment and predict how the situation might develop and how variables might interact (Brehmer & Dörner, 1993; Güss et al., 2011). Self-reflection should also reduce error caused by bias, because, when decision makers engage in self-reflection, they slow down and think about their knowledge of the situation and the relevance of their knowledge (Güss et al., 2009). Self-reflective decision makers are more likely to question the accuracy of heuristics and their inferences and recognize limitations of what they know (Dodson & Schacter, 2002; Winne & Nesbit, 2010).
Decision makers formulate a strategy within the scope of their ability and knowledge and adjust their strategy as they work through a DDM task. Decision makers may err if they take aggressive actions without developing a proper strategy or if they do not recognize and then correct for the system’s dynamics (e.g., cyclic changes such as seen in business cycles: Grobler, Milling & Thun, 2008). Self-reflective and strategic questioning promotes awareness and strategic flexibility because it forces decision makers to evaluate their decisions in light of their learning and alternative strategies.
Evaluation of outcome equates with error management. Self-reflection in this step forces decision makers to differentiate the effects of their actions from the autonomous development of a system (Schaub, 2007). It can also clarify how the effects of implemented decisions propagate through a system over time. Accordingly, decision makers who regularly self-reflect on feedback should have a more accurate idea of progress in relation to their goals, a more comprehensive understanding regarding the appropriateness of their strategies, and strategic control in pursuit of their goals (Locke & Latham, 2002; Osman, 2010). Trainings in DDM should stress the importance of conducting error management and encourage decision makers to ask reflective questions, gather additional information, and elaborate before formulating and acting on an alternative plan.
From our discussions on the potential value of using training to promote self-reflection during DDM, and considering individual differences in self-reflection, we make the following two predictions. A training program in dynamic decision making strategies that promotes self-reflection will allow participants (1) to react with more sensitivity to the demands of the situation and (2) to ultimately perform better than untrained individuals. To specify these predictions, the simulation CHOCO FINE will be briefly described and specific strategic behavior patterns will be discussed.
CHOCO FINE is a computer simulation of a chocolate producing company in Vienna. Working with CHOCO FINE, every participant takes the role of CEO and manages production, marketing, and sales within the virtual company. The simulation can be described as a top management game or complex simulation. It was originally developed in 1993 at University of Bamberg in Germany through collaboration of Dietrich Dörner and experts within the business field (Dörner & Gerdes, 2001). The current study used a revised version (2003) of the simulation, which contains more than 1,000 simulated variables. The European Center for the Development of Vocational Training (Cedefop) and the Federal Institute for Vocational Education and Training (Germany, BIBB) endorsed CHOCO FINE as a valid training system for complex and dynamic work-related situations where decision making and action are required. Preliminary studies in the United States (N = 150) were conducted by the second author to determine whether CHOCO FINE is a valid instrument in the US (Güss, Edelstein, Badibanga & Bartow, 2015). Even though overall profit declined for all groups, results validated CHOCO FINE as an instrument because performance followed the expected trend: performance operationalized as account balance was highest for US business owners, followed by US undergraduate business students, and lowest for US undergraduate psychology students.
The participants’ main task is to increase profit for the company. Participants have complete strategic freedom because CHOCO FINE does not require any actions in order to progress through the months other than simply clicking “Continue” at the bottom of the computer screen. If participants decide to progress to a subsequent month without making changes to the system (e.g., they cannot decide what to do), implemented decisions will remain in effect. Monthly financial gains and losses to are automatically displayed. Information that is not conveyed automatically (e.g., monthly expenditures on raw materials, whole sale prices for the different types of chocolate) is displayed when the related command is clicked. The program stores every decision each participant makes in external files, which allows for analysis of DDM results and strategies.
CHOCO FINE has three screen windows that participants can easily navigate among. The main screen (1) shows for example information regarding costs, sales, production, orders, raw materials, and account balance. The production screen (2) shows for example information regarding the six machines, their capacities, and which of the eight chocolates are produced on which machine for each day of the month. Participants can also implement changes in production on this screen. The marketing screen (3) shows for example the city map and the different districts. For each of the 23 districts, a pie graphs provide information about the size of the local market and the market shares of the 5 competitors. The marketing screen offers menus from which participants can gather additional information and make decisions regarding for example advertising, delivery, prices, product profile or customers’ profiles.
Why would a decision-making training and why would high self-reflection be related to better performance in CHOCO FINE? We attempt now to define decision-making behaviors and strategies that differentiate participants who underwent a training from participants in the control group. Similarly, we attempt to define decision-making behaviors and strategies that differentiate self-reflective participants from participants less inclined to self-reflect. The expectations described here are based in part on our familiarity with the simulation; each one can also be seen as a question that we address.
We asked whether decisions in three key areas (i.e., expenses for information collection, expenses for advertising, and number of representatives) differed from the decisions made in the previous month (coded as 0 for no change in respective area and 1 for a change) and then summed up these changes over the first 8 months for each area and overall.
We assessed the time spent for the first month in the CHOCO FINE simulation. More time spent can be regarded as an indicator for deeper processing of the situation. Participants proceeded to the next month of the simulation by clicking the “Continue” button whenever they felt they made enough decisions. Participants took on average 17 minutes to complete the first month of CHOCO FINE.
We also assessed another variable that shows consistent planning or mindfulness in decision making. We calculated the difference from month to month in advertising expenses and calculated the mean for the 8 months.
At the beginning of the CHOCO FINE simulation, i.e., in month 1 and 2, there is a special demand for two kinds of chocolates, Nuts and Bitter. This demand is not met. Sales for these two kinds of chocolates could be much higher as their orders indicate. Their high orders are shown with bar graphs on the computer screen for the participants. There can be several causes for the low sales of these two chocolates, for example, poor organization of production, inadequate advertising, or improper product distribution. A low deviation of sales from orders (averaged for the first two months) would indicate successful problem solution.
In month 5, an announcement appears on the computer screen. It indicates that one competitor launches an advertisement campaign emphasizing quality and environmental friendliness of its products.
Participants start with 2,182,000 US$. A red bar graph on the main computer screen provides the participants with their monthly account balance. We looked at account balance in month 2 and selected all participants who had less than 2,000,000US$. An amount of less than 2 Million US$ can be regarded as a psychological threshold. Only 13 of the 65 participants had an account balance greater than 2,000,000US$ at the end of month 2. We than looked at the 52 remaining participants and analyzed how many changes they made in three key areas (information collection, advertising, and representatives) in response to the feedback displayed as account balance.
Additionally, other decision-making behaviors could be indicative of effects of training and self-reflection. These variables were means for month 1 to 8 for number of representatives, expenses for information collection, expenses for advertising, production numbers, order numbers, and sales numbers.
To sum up, training DDM would correlate with breadth of decision making and adjusting failing strategy. High self-reflection would also correlate with adjusting failing strategy, but also correlate with time spent at the beginning of the simulation (deep processing), mindfulness/consistency in planning, and problem sensitivity and successful problem solving. No effects are expected for other outcome variables.
Participants were 69 students recruited from graduate business courses in the College of Business at a University in the southeastern of the United States. Graduate business students were selected because they have necessary background knowledge to perform well in a highly complex business simulation. Participants’ ages ranged from 22 to 58 years (M = 29.47, SD = 6.68). 43% of participants were female and 57% were male. The sample’s ethnic and gender distribution was similar to the distribution of the university’s graduate population, with 87% being Caucasian. The experimental group consisted of 38 students who received DDM training. The control group consisted of 31 participants, who did not receive training. Participants were assigned to either the experimental/training group or the control group based on their enrollment in one of two sections of an MBA course. Both sections were night classes taught by the same professor. Sections did not differ in age, gender, SES, or computer experience. Two participants from each group were excluded from the analysis because of technical problems with their PCs and the saved data sets.
Both the training and the non-training control groups immediately received three pages instructions about CHOCO FINE including explanations regarding key variables and screen shots of the main screen (1), the production screen (2), and the marketing screen (3). (See http://journal.sjdm.org/14/14411/Surveys.pdf.) Every participant kept the instructions throughout the experiment.
Approximately half of the participants partook in a brief, experimenter-led training that taught about self-reflection in the context of DDM (http://journal.sjdm.org/14/14411/Presentation.pdf and http://journal.sjdm.org/14/14411/NOTEStoSlides.pdf.). The time for training was 10 minutes. The training used a PowerPoint presentation (8 slides) displayed on an overhead projector in a classroom equipped with 50 computers to educate students in the DDM process and it provided participants with an aid for carrying out self-reflection. The training explained DDM by breaking the DDM process down into its steps: Goal identification; Information gathering; Elaboration and prediction; Planning; Decision making and action; Effect control and Self-evaluation. The presentation included one slide for each DDM step and the concluding slide showed all of the steps together. In addition to defining each step, the experimenter also provided one business application for each step. Using a familiar business context should have helped business students incorporate the DDM steps into existing schemas.
The experimenter used caution to ensure that the training examples did not suggest any specific actions that could influence participants’ decision-making behaviors and strategies in CHOCO FINE during the second-half of the experiment. Listing the DDM process as a linear progression of steps facilitates comprehension but the DDM process is cyclic, and self-reflection occurs not only during “evaluation of outcome” but during the other steps as well. The experimenter clarified this as part of the training and further explained that self-reflection increases situational awareness and may lead to insight, which can then be applied to redefine goals, gather information, and so forth. Additionally, the experimenter gave participants a handout of the DDM process with three self-reflective questions similar to those used by Güss et al. (2009): What did I do well? What can I do better? How can I use the decision-making steps more effectively? These aids were discussed under the last step “Effect Control and Self-evaluation” and were expected to increase participants’ self-reflection while they worked on the complex and dynamic business simulation.
The CHOCO FINE simulation as well as the decision-making behaviors and strategies and outcome variables were described in the introduction section.
Grant et al. (2002) developed the Self-Reflection and Insight Scale (SRIS), which incorporates three factors in the self-regulation cycle: need for self-reflection (e.g., “It is important to me to try to understand what my feelings mean”), engagement in self-reflection (e.g., “I frequently take time to reflect on my thoughts”), and insight (e.g., “I usually know why I feel the way I do”). In the current study, the inter-item reliability was high for the total 20-item measure and each of the 3 subscales (after reverse-scoring the appropriate items): SRIS (20 items; Cronbach’s alpha = 0.85); need for self-reflection (6 items; Cronbach’s alpha = .79); engagement in self-reflection (6 items; Cronbach’s alpha = .76); insight (8 items; Cronbach’s alpha = .75). Additionally, need for self-reflection correlated positively with engagement in self-reflection (r = .75, p = .000), and the two self-reflection subscales combined correlated positively with the insight subscale (r = .26, p = .04). Insight is “the clarity of understanding one’s thoughts, feelings, and behaviors” (Grant et al., 2002, p. 821), but insight items mostly refer to one’s own feelings. The significant but weaker correlation is due to the fact that while people achieve insight through self-reflection, self-reflection does not always lead to insight. We used the overall mean score of the 20 items (Likert Scale from 1-7) for our analyses. Higher scores on the SRIS reflect purposeful, self-regulatory behaviors directed towards goal attainment (Grant et al., 2002). The items are in http://journal.sjdm.org/14/14411/Surveys.pdf (called “Personality Questionnaire”).
A brief demographic survey (also in http://journal.sjdm.org/14/14411/Surveys.pdf) was also administered to assess for example age, gender, major, and computer experience. (Every participant showed extensive experience with computer programs and usage of the mouse.)
In the experimental condition, 38 of the 69 participants participated in a 10 minute experimenter-led training in DDM and received a handout outlining the decision-making process and three self-reflective questions. These participants kept the handout while managing CHOCO FINE to aid them with self-reflection as they worked the simulation. These participants were encouraged to ask questions about the training, but they asked only a few questions seeking clarification. The remaining 31 participants served as controls and did not receive training or training materials. Participants in both the trained and non-trained groups were asked to work on the CHOCO FINE simulation. All 69 participants received a three page overview of CHOCO FINE with screen shots of the three main screens and instructions. The instructions outlined the locations of specific information, the costs associated with various actions, and the interpretation of graphs and other visuals. Participants individually managed CHOCO FINE for a minimum of 45 minutes and completed at least 8 months within the simulation. Most participants completed 8 months within a range of 45 to 70 minutes. After ending the simulation, participants took 5 to 10 minutes to complete the SRIS (Grant et al., 2002) and answer questions regarding demographics and computer experience.
Figure 1: Account balance in the eight months of CHOCO FINE for the trained and highly reflective participants, trained and less reflective participants, untrained and highly reflective participants, and the untrained and less reflective participants. 
Table 1: Results of t-tests and descriptive statistics (when Levene’s test for equality of variances was violated the t-statistic not assuming homogeneity of variance is reported).
Training group No-training group Outcome M SD M SD d Self-reflection SRIS 4.51 .72 4.55 .59 -.07 Performance (Balance, mean of 8 mo., millions) 1.66 .37 1.22 .56 .97 Breadth of decision making and changes: Total changes - Information 2.64 2.19 4.15 1.95 -.73 Total changes - Advertising 7.31 1.24 3.58 1.60 2.67 Total changes - Representatives 6.14 2.54 2.88 1.63 1.59 Total changes - Overall 16.08 3.77 10.62 3.59 1.48 Depth of processing/ Mindfulness: Time for first month 5,054.54 5,401.72 4,334.54 5,155.89 .13 Consistency in advertising 985.87 550.17 1,116.64 677.30 -.02 Problem sensitivity: Nuts problem: Orders minus sales 113.04 43.79 146.37 58.57 -.66 Bitter problem: Orders minus sales 38.11 41.77 58.23 63.09 -.39 Changes to competitor announcement 2.03 .65 1.92 1.16 .14 Adjusting failing strategy: Changes in month 3 1.81 .69 1.50 .93 .41 Other outcome variables: Representatives average 15.95 7.09 23.33 16.95 .75 Advertising average 23,439.88 23,196.32 19,940.04 24,422.41 .15 Info average 4072.92 3894.75 3096.98 3123.15 .28 Production average 508.60 78.54 554.25 38.18 -.83 Orders average 549.57 223.09 604.12 210.31 -.25 Sales average 310.90 110.02 321.87 93.36 -.11
Table 2: Results of Pearson correlations and descriptive statistics.
M SD Self-reflection Performance Self-reflection SRIS 4.53 .67 1.00 .23 Performance (Account balance, mean of 8 months, in millions) 1.46 .51 .23 1.00 Breadth of decision making and changes: Total changes - Information 3.27 2.21 -.15 -.19 Total changes - Advertising 5.74 2.32 -.06 .40 Total changes - Representatives 4.77 2.72 -.10 .22 Total changes - overall 13.79 4.57 -.16 .24 Depth of processing/ Mindfulness: Time for first month 1041.92 606.22 .17 -.08 Consistency in advertising 4,733.31 5,264.72 -.24 .05 Problem sensitivity: Nuts problem: Orders minus sales 127.33 52.89 .10 -.24 Bitter problem: Orders minus sales 46.73 52.49 -.06 .02 Changes to Competitor announcement 1.98 .90 -.22 .07 Adjusting failing strategy: Changes in month 3 1.66 .82 -.03 .20 Other outcome variables: Representatives Total average 19.05 12.65 -.14 -.39 Advertising Total average 21,884.40 23,619.69 -.37 -.20 Info Total average 3,637.50 3,577.99 .12 -.12 Production Total average 528.16 67.91 -.05 -.04 Orders Total average 572.95 217.68 -.11 .17 Sales Total average 315.60 102.56 -.07 .16
To illustrate the results in Figure 1, we conducted a median split (M = 4.53, Median = 4.55) to classify participants as “high” versus “low” in self-reflection. The low self-reflection group had a mean of 3.99 (SD = .42, n = 32), the high self-reflection group a mean of 5.08 (SD = .32, n = 33). The self-reflection scores for the trained group (M = 4.51, SD = .72, n = 36) did not differ significantly from the scores of the untrained group (M = 4.55, SD = .59, n = 29), t(63) = −.26, p = .80. Being aware of the limitations of median or mean splits, further statistical analyses will use independent-samples t-tests to compare the training and non-training control group (Table 1) and Pearson correlations to investigate the relationship between self-reflection with performance and decision-making strategies and other variables (Table 2).
Of particular interest, in an analysis of variance, the linear slope of earnings over the 9 months depended on both training (t(62) = 4.38, p < .001) and and self-reflection score (t(62) = 2.34, p = .022); in a separate analysis, the interaction was not significant.
We predicted, as previously described, that training in task understanding and DDM would correlate with breadth of decision making and adjusting failing strategy. High self-reflection would also correlate with adjusting failing strategy, but would also correlate with time spent at the beginning of the simulation (deep processing), mindfulness/consistency in planning, and problem sensitivity and successful problem solving. No effects were expected for other outcome variables.
Results in Table 1 confirm our expectation that training would lead to more breadth in decision making and more changes. With the exception of changes in information collection, changes in advertising, representatives, and overall were significantly higher in the training group compared to the control group. The effect sizes Cohen’s d were all higher than .8 and can be regarded as large (Cohen, 1988). This means that those participants who received training, more often changed decisions in these domains.
Contrary to our expectations, participants in the training condition did not adjust their failing strategy more often compared to the control group, but they were more sensitive to the “Nuts chocolate” problem, being able to lower the difference between orders and sales significantly.
Although we did not expect any differences in the other outcome variables, participants in the training group hired fewer representatives and had lower production numbers. They did not differ in expenses for advertising, expenses for information collection, total orders, and total sales.
Table 2 presents the correlations between decision-making strategies, other variables, and self-reflection. In addition the table shows the correlation of these variables with performance, defined as the mean account balance of the 8 months. Descriptive statistics of the variables are presented as well.
We expected high self-reflection to correlate with adjusting failing strategy, with time spent at the beginning of the simulation (deep processing), mindfulness/consistency in planning, and, with problem sensitivity and successful problem solving. No effects were expected for other outcome variables.
We found three significant correlations between the strategies, variables, and self-reflection. The first one was between self-reflection and advertising costs. High self-reflection was related to lower advertising expenses. The second one was between mindfulness/consistent planning and self-reflection scores, indicating that higher self-reflection scores were related to lower average deviation in advertising from month to month. In other words, there was more consistency in advertising expenditures across the months for participants with higher self-reflection scores. The third finding was an almost-significant negative correlation between changes made after a competitor launches an advertisement campaign. We come back to this result in the discussion section.
It is noteworthy that several of the defined strategies correlate positively with performance: Changes in advertising, changes overall, adjusting failing strategy for Nuts chocolate, and low average number of representatives.
It is not the amount of money spent for advertising or information collection that predicts performance. It is also not the mere number of produced products, orders or sales that predicts performance. It is the adjustment to the changes in the market that predicts performance.
We have shown that training in decision-making strategies and high self-reflection are related to better performance in the CHOCO FINE business simulation. Trained participants completed the simulation with a larger account balance than untrained participants. High self-reflection participants also ended the simulation with a higher account balance than low self-reflection participants.
We conducted an in-depth analysis of dynamic decision-making strategies and investigated why training and high self-reflection would lead to better performance. We expected training in task understanding and decision making to correlate with breadth of decision making and adjusting failing strategy. We expected high self-reflection also to correlate with adjusting failing strategy, but additionally to correlate with time spent at the beginning of the simulation (deep processing), mindfulness/consistency in planning, and problem sensitivity and successful problem solving.
Results do show more changes in key areas in the training group compared to the non-training group. These changes also correlated positively with overall performance. The training group also solved the initial nuts chocolate problem more successfully than the control group. This variable also correlated significantly with performance. The significant correlations of decision-making strategies with performance speak for the validity of the operationalized decision-making strategies and errors. In sum, trained participants compared to untrained participants performed better in the DDM task, made more adjustments and changes, and showed more sensitivity and problem-solving skills for initial problems of the company.
Two of the strategies and variables, consistency in advertisement and few changes as a result to competitor announcement, correlated significantly with self-reflection survey scores. The finding related to self-reflection and few changes is counterintuitive. Why is there a negative correlation between self-reflection scores and changes made after a competitor launches an advertisement campaign? Why would participants scoring high on self-reflection make fewer changes after this information was presented on the screen? One possible explanation might be that high self-reflective participants realize that the competitor is only a small competitor for CHOCOFINE, in fact the third largest competitor out of five. There are two much more important competitors. Second, participants with high self-reflection scores might not be ‘thrown off’ as easily as participants with lower self-reflection scores. They might stick to the strategy they previously decided on. The negative correlation between total changes made and self-reflection scores (r = −.16) validates this argument. The more consistent planning in advertisement with fewer dramatic changes also validates the fewer changes as a reaction to the competitor announcement. Third, participants with high self-reflection scores might be aware of their own product profile, own customers and competitors after working on four simulated months and therefore not need to make changes as a result of the announcement. The positive correlation between time spent on the first month and self-reflection scores (r = .17) might be an indicator for in-depth processing and seeking of relevant information at the beginning of the simulation. In brief, high self-reflection in CHOCO FINE does not mean adapting to all changes in the environment, but means knowing when to adopt and when not.
Another interesting finding is that both training and high self-reflection seem to make the decision maker more cautious. The training group hired fewer representatives and had lower production numbers. High self-reflection correlated with fewer expenses for advertising. Perhaps participants through training and high self-reflection become more aware about what it takes to keep the system variables in a “balance”. A key difference between training condition and self-reflection, however, is that training compared to no training is related to making many changes and high self-reflection is related to making fewer changes.
The training with special focus on self-reflection was designed to improve DDM and to reduce some of the challenges and frustrations associated with the DDM task. Since previous studies (e.g., Campitelli and Labollita, 2010; Grant et al., 2002; Locke & Latham, 2006; Sanders & McKeown, 2008) suggested self-reflection plays a fundamental role in DDM, we also measured individual differences in self-reflection and included self-reflective aids as a part of the training. The aids were designed to motivate participants to use self-reflection, increase self-efficacy for using self-reflection, and increase self-efficacy in the DDM task. The results of this experiment are notable because the training satisfied Dörner’s (1996) call for a brief and low-cost training that improves management performance. Looking at the decision-making process, results suggest that the training served this intended purpose. The differences in account balance between trained and untrained participants increased as trained participants lost less money than untrained participants each subsequent month. This trend in the data implies that the trained participants increased their understanding of the simulation more than the untrained participants as they progressed through months in the simulation.
Additionally, similar top management games (e.g., Ramnarayan et al., 1997) associated poor performance with errors that are negatively related to self-reflection (e.g., inadequate effect control) not with insufficient managerial knowledge or cognitive limitations. Other research using similar participant populations and comparable simulations also showed that performance was related to cognitive errors and biases in decision-making strategies (e.g., Dörner, 1996; Güss & Dörner, 2011; Ramnarayan et al., 1997). Also in the current study, errors (e.g., lack of problem sensitivity, lack of changing strategy and adjusting failing strategy) were associated with lower performance. The results of this experiment extend previous research by analyzing decision-making strategies in depth and relating them to training, self-reflection, and performance.
Due to the complexity and novelty of CHOCO FINE, account balance decreased for all groups. The challenges presented in CHOCO FINE were difficult for participants to overcome and losing money was certainly frustrating for participants (see Starker, 2012, for the role of emotions during DDM in CHOCO FINE). However, all the necessary information for participants to perform well in CHOCO FINE was available in the instructions and on the screen, and 11% of all participants ended the simulation with an account balance of more than $2 million. The graphs look skewed because some participants lost a lot of money.
One limitation of this study is that self-reflection was assessed via self-report. Although there was a significant relationship between self-reflection and performance, we cannot tell exactly when and how participants engaged in self-reflection during the CHOCO FINE simulation. The field of DDM would benefit from continued research on the influence of self-reflection. Future research could further investigate the validity of the SRIS self-report self-reflection instrument and relate it to actual self-reflection in dynamic decision making assessed for example in think-aloud protocols or in answers to provided self-reflection questions (e.g., Güss et al., 2010).
A student population limits generalizability. However, demographic survey results showed that approximately 80% of the participants have worked above an entry level position in their company. Still, student data may not reflect the decisions managers make in organizational contexts where they are held accountable for the outcomes of their decisions (see Güss et al., 2015). Student participants may take more risks or may not be motivated to utilize their cognitive resources in a simulation where they are not as responsible for their performance as in real world settings. However, as experimenters, we observed participants and noted that they played CHOCO FINE in earnest and some of them did not even want to stop working on the simulation.
The training explained that breaking a main goal down into sub-goals might facilitate progress toward the main goal. The training also encouraged participants to self-reflect on their progress in relation to their goals. However, the training did not explain how to define adequate sub-goals. Performance feedback in CHOCO FINE informed participants on progress toward their main goal of obtaining profit, but not how individually determined sub-goals affected profit (e.g., how a marketing campaign affects profit). Future DDM trainings could provide decision makers with meta-cognitive aids that help them define appropriate sub-goals, and ultimately make decisions that bring them closer to achieving their main performance goals (Locke & Latham, 2002; Osman, 2010).
High self-reflection and training participants in the steps of dynamic decision making can improve performance in a dynamic and complex task like the chocolate company CHOCO FINE. A prototypical person who utilizes self-reflection compared to a prototypical person showing little self-reflection performed better, spent more time at the beginning to analyze the problem situation, made fewer changes, was not as easily thrown off by unexpected events, showed fewer drastic changes and more consistency in decision-making strategy.
The prototypical person who underwent a training program on the steps in dynamic decision making compared to a person who did not receive such a training program, performed better, made more changes in decisions, showed more breadth in their decisions and more sensitivity to current problems.
The results of the present study carry practical applications for managers who make decisions in stressful, complex, and dynamic work environments. Organizations may benefit if they encourage self-regulatory decision making. A short training in DDM that reinforces self-reflection may lead to more successful decision making. Organizations could establish and promote a culture that values time set aside for self-reflection on decision-making steps. Self-reflection can occur in a very short break, and its influence on behavior should ultimately have a positive impact at the organizational level.
BIBB (Federal Institute for Vocational Education and Training - Germany) (2005). Strategic flexibility, a vital tool for today’s specialists and managers. Last modified Feb 9, 2005 originally posted 11.12.2003 http://www.bibb.de/en/8494.htm.
Brehmer, B. (1992). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81, 211–241. http://dx.doi.org/http://dx.doi.org/10.1016/0001-6918(92)90019-A.
Brehmer, B., & Dörner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits if the laboratory and the deep blue sea of field study. Computers in Human Behavior, 9, 171–184. http://dx.doi.org/10.1016/0747-5632(93)90005-D.
Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with judgments and choices. Judgment and Decision Making, 5, 182–191.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Dörner, D. (1996). The logic of failure: Recognizing and avoiding error in complex situations. New York, NY: Metropolitan Books.
Dörner, D., & Gerdes, J. (2003). SchokoFin. Computerprogramm [Choco Fine. Computer program]. Institut für Theoretische Psychologie, Universität Bamberg, Germany.
Dodson, C. S., & Schacter, D. L. (2002). When false recognition meets metacognition: The distinctiveness heuristic. Journal of Memory and Language, 46, 782–803. http://dx.doi.org/10.1006/jmla.2001.2822.
European Centre for the Development of Vocational Training (CEDEFOP) http://www.cedefop.europa.eu.
Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. The Journal of Problem Solving, 4, 19-42. http://dx.doi.org/10.7771/1932-6246.1118.
Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218–233.
Frensch, P., & Funke, J. (Eds.) (1995). Complex problem solving: The European perspective. Hillsdale, NJ: Erlbaum.
Funke, J. (2003). Problemlösendes Denken [Problem-solving Thinking]. Stuttgart, Germany: Kohlhammer.
Funke, J. (2010). Complex problem solving: A case for complex cognition. Cognitive Processing, 11, 133–142. http://dx.doi.org/10.1007/s10339-009-0345-0.
Gonzalez, C. (2004). Learning to make decisions in dynamic environments: Effects of time constraints and cognitive abilities. Human Factors, 46, 449–460. http://dx.doi.org/10.1518/hfes.46.3.449.50395.
Grant, A. M., Franklin, J., & Langford, P. (2002). The self-reflection and insight scale: A new measure of private self-consciousness. Social Behavior and Personality, 30, 821–836. http://dx.doi.org/10.2224/sbp.2002.30.8.821.
Grobler, A., Milling, P. M., & Thun, T. (2008). System dynamics as a structural theory in operations management. Production and Operations Management, 17, 373–384. http://dx.doi.org/10.3401/poms.1080.0023.
Güss, C. D. (2011). Fire and ice: Testing a model on cultural values and complex problem solving. Journal of Cross-Cultural Psychology, 42, 1279–1298. http://dx.doi.org/10.1177/0022022110383320.
Güss, C. D., & Dörner, D. (2011). Cultural differences in dynamic decision-making strategies in a non-linear, time-delayed task. Cognitive Systems Research, 12, 265–376. http://dx.doi.org/10.1016/j.cogsys.2010.12.003.
Güss, C. D., Edelstein, H. D., Badibanga, J., & Bartow, S. (2015). The flexibility of experts in dynamic business decision making. Manuscript submitted for publication.
Güss, C. D., Evans, J., Murray, D., & Schaub, H. (2009). Conscious versus unconscious processing in dynamic decision making tasks. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53, 227–231. http://dx.doi.org/10.1177/154193120905300414.
Güss, C. D., Tuason, M., & Gerhard, C. (2010). Cross-national comparisons of complex problem-solving strategies in two microworlds. Cognitive Science, 34, 489–520. http://dx.doi.org/10.1111/j.1551-6709.2009.01087.x.
Helsdingen, A. S., van den Bosch, K., van Gog, T., & van Merriënboer, J. J. G. (2010). The effects of critical thinking instruction on training complex decision making. Human Factors, 52, 537–545.
Klein, G. (1998). Sources of power: How people make decisions. Cambridge, MA: MIT Press.
Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57, 705–717. http://dx.doi.org/10.1037/0003-066X.57.9.705.
Locke, E. A. & Latham, G.P. (2006). New directions in goal setting theory. Current Directions in Psychological Science, 15, 265–268. http://dx.doi.org/10.1111/j.1467-8721.2006.00449.x.
Osman, M. (2010). Controlling uncertainty: a review of human behavior in complex dynamic environments. Psychological Bulletin, 136, 65–86. http://dx.doi.org/10.1037/a0017815.
Osman, M. (2011). The effects of externally set or self set goals on learning in an uncertain environment. Learning and Individual Differences.(available online October 6, 2011—in press) http://dx.doi.org/10.1016/j.lindif.2011.09.012.
Ramnarayan, S., Strohschneider, S., & Schaub, H. (1997). The trappings of expertise and the pursuit of failure. Simulation and Gaming, 28, 28–44. http://dx.doi.org/10.1177/1046878197281004.
Sanders, R. L., & McKeown, L. (2008). Promoting reflection through action learning in a 3D virtual world. International Journal of Social Sciences, 2, 50–58.
Schaub, H. (2001). Problemlösen und Persönlichkeit [Problem solving and personality]. Weinheim, Germany: Beltz.
Schaub, H. (2007). The importance of the characteristics of the task to understand team mental models. CoDesign, 3, 37–42. http://dx.doi.org/10.1080/15710880601170800.
Snyder, M. (1974). Self-monitoring of expressive behavior. Journal of Personality and Social Psychology, 30, 526–537. http://dx.doi.org/10.1037/h0037039.
Starker, U. (2012). Emotionale Adaptivität [Emotional adaptivity]. Lengerich: Pabst.
Sternberg, R. J. (1986). Intelligence applied? Understanding and increasing your intellectual skills. San Diego,CA: Harcourt Brace Jovanovich.
Winne, P. H., & Nesbit, J. C. (2010). The psychology of academic achievement. Annual Review of Psychology, 61, 653–678. http://dx.doi.org/10.1146/annurev.psych.093008.100348.
Copyright: © 2015. The authors license this article under the terms of the Creative Commons Attribution 3.0 License.
This document was translated from LATEX by HEVEA.