\documentclass[twocolumn]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage{amsmath,amsthm,amssymb,amsfonts,newtxtext,newtxmath} 
\usepackage{array,graphicx,dcolumn,multirow,hevea,abstract,hanging}
\usepackage[labelfont=sc,textfont=sf]{caption}
\usepackage[hyperfootnotes=false,breaklinks=true]{hyperref} % was dvipdfmx
% \usepackage{apacite} % must come afer hyperfootnotes
% \setlength{\bibleftmargin}{1em}
% \setlength{\bibindent}{-1em}
\urlstyle{rm}
\usepackage[hyphenbreaks]{breakurl}
\usepackage{booktabs} % \toprule \midrule \bottomrule \cmidrule(lr){a-b}
% define centered and ragged columns:
\newcolumntype{L}[1]{>{\raggedright\arraybackslash }p{#1}} % can use m{}
\newcolumntype{C}[1]{>{\centering\arraybackslash }p{#1}}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash }p{#1}}
\newcolumntype{d}[1]{D{.}{.}{#1}} % d{3.2} for 3 places on l, 2 on r
\newcommand{\mc}{\multicolumn}
\topmargin=-.3in \oddsidemargin=-.1in \evensidemargin=-.1in \textheight=9in \textwidth=6.8in
\setlength\tabcolsep{1mm}
\setlength\columnsep{5mm}
\setlength\abovecaptionskip{.5ex}
\setlength\belowcaptionskip{.5ex}
\setlength\belowbottomsep{.3ex}
\setlength\lightrulewidth{.04em}
\renewcommand\arraystretch{1.2}
% \renewcommand{\baselinestretch}{1.00} \large\normalsize
\widowpenalty=1000
\clubpenalty=1000
\setlength{\parskip}{0ex}
\let\tempone\itemize
\let\temptwo\enditemize
\let\tempthree\enumerate
\let\tempfour\endenumerate
\renewenvironment{itemize}{\tempone\setlength{\itemsep}{0pt}}{\temptwo}
\renewenvironment{enumerate}{\tempthree\setlength{\itemsep}{0pt}}{\tempfour}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\setcounter{page}{1} % start with first page

\title{Kenneth R. Hammond’s contributions to the study of judgment and decision making}

\author{
Mandeep K. Dhami\thanks{Department of Psychology, Middlesex University,
 The Burroughs, Hendon, London NW4 4BT, UK. E-mail: m.dhami@mdx.ac.uk.}
\and
Jeryl L. Mumpower\thanks{Bush School of Government and Public Service, Texas
A\&M University, College Station, Texas, USA.}
}

\date{} % leave empty
\begin{document} % goes here

% change next 3 lines each issue
\newcommand{\jref}{http://journal.sjdm.org/vol13.1.html}
\newcommand{\jhead}{Judgment and Decision Making, Vol.~13, No.~1}
\newcommand{\jdate}{January 2018}
\pagestyle{myheadings} \markright{\protect\small \href{\jref}{\jhead},
  \jdate \hfill Kenneth R. Hammond's contribution to JDM \qquad}
\begin{htmlonly}
\href{\jref}{\jhead}, \jdate, pp.\ 1-22
\end{htmlonly}
%\begin{latexonly}
\twocolumn[
\vspace{-.5in}
{\small \href{\jref}{\jhead}, \jdate, pp.\ 1--22}
%\end{latexonly}

\maketitle

%\begin{latexonly}
\vspace{-3mm}
\begin{onecolabstract}
%\end{latexonly}

Kenneth R. Hammond (1917--2015) made several major contributions to the
science of human judgment and decision making. As a student of Egon
Brunswik, he kept Brunswik’s legacy alive – advancing his theory of
probabilistic functionalism and championing his method of
representative design. Hammond pioneered the use of Brunswik’s lens
model as a framework for studying how individuals use information from
the task environment to make clinical judgments, which was the
precursor to much `policy capturing’ and `judgment analysis’ research.
Hammond introduced the lens model equation to the study of judgment
processes, and used this to measure the utility of different forms of
feedback in multiple-cue probability learning. He extended the scope of
analysis to contexts in which individuals interact with one
another – introducing the interpersonal learning and interpersonal
conflict paradigms. Hammond developed social judgment theory which
provided a comprehensive quantitative approach for describing and
improving judgment processes. He proposed cognitive continuum theory
which states that quasi-rationality is an important middle-ground
between intuition and analysis and that cognitive performance is
dictated by the match between task properties and mode of cognition.
Throughout his career, Hammond moved easily from basic laboratory work
to applied settings, where he resolved policy disputes, and in doing
so, he pointed to the dichotomy between theories of correspondence and
coherence. In this paper, we present Hammond’s legacy to a new
generation of judgment and decision making scholars.

\smallskip
\noindent
Keywords: lens model, policy capturing, cognitive feedback,
interpersonal learning, interpersonal conflict, social judgment theory,
cognitive continuum theory
%\begin{latexonly}
\end{onecolabstract}\bigskip
]
%\end{latexonly}

{\renewcommand{\thefootnote}{}

\footnotetext{



Copyright: \copyright\ 2018.
The authors license this article under the terms of the
\href{http://creativecommons.org/licenses/by/3.0/}{Creative Commons
  Attribution 3.0 License.}
}}

\saythanks

\section{Introduction}

Kenneth R. Hammond [1917--2015] is perhaps one of the most important
figures in the history of the psychology of human judgment and decision
making. Although his influence has already been considerable, we
anticipate that it will grow over time because his contributions build
on a coherent and comprehensive theoretical framework, which lays a
foundation for the development of a cumulative body of theory and
research. It may be challenging, however, for future generations of
scholars to access the breadth and depth of Hammond’s work, which
spanned over seven decades, and is documented in over 100 articles,
seven authored books, and five edited volumes. In this paper, our goal
is to review and synthesize Hammond’s major contributions to the study
of human judgment and decision making.

Hammond made at least eight major contributions that we will discuss.
First, he virtually single-handedly kept Egon Brunswik’s legacy alive
in psychology (Hammond, 1966; Hammond \& Stewart, 2001b). Hammond built
on Brunswik’s theory of probabilistic functionalism and his lens model,
and he championed Brunswik’s method of representative design. Second,
Hammond pioneered the use of Brunswik’s lens model as a framework for
studying how individuals use information from the task environment to
make clinical judgments (Hammond, 1955). This was the precursor to what
is known today as `policy capturing’ and `judgment analysis’ research.
Third, Hammond formulated the lens model equation which provided a
quantitative tool for modeling and analyzing both the task environment
side and human judgment side of the lens model for studying judgment
processes (Hammond, Hursch \& Todd, 1964; Hursch, Hammond \& Hursch,
1964). Fourth, Hammond employed this equation when examining
multiple-cue probability learning, making contributions regarding the
effects of different forms of feedback on learning processes (Hammond
\& Summers, 1965; Todd \& Hammond, 1965). Fifth, Hammond extended the
application of the lens model to contexts in which individuals
interacted with one another – introducing both the interpersonal learning
paradigm (Hammond, Wilkins \& Todd, 1966a) and the interpersonal
conflict paradigm (Hammond, 1965; 1972; 1973; Hammond, Todd, Wilkins,
\& Mitchell, 1966b). Sixth, Hammond and his colleagues developed social
judgment theory (SJT), which integrated prior work involving
the lens model into a comprehensive quantitative approach for
describing judgment processes and exploring means for improving
cognitive performance (Hammond, Stewart, Brehmer \& Steinmann, 1975).
Seventh, grounded in, but going beyond Brunswikian concepts and SJT,
Hammond developed cognitive continuum theory (Hammond, 1996a; 2000a;
2001a; Hammond, Hamm, Grassia \& Pearson, 1987). He proposed that
cognition moves along an intuitive-analytical continuum, with
quasi-rationality as a middle-ground, and that cognitive tasks induce
different modes of cognition, implying that cognitive performance is
dictated by the match between properties of the task and mode of
cognition. Finally, Hammond moved seemingly effortlessly from
conducting basic work in the laboratory to solving applied
policy-related problems (Hammond, 1996a; 2007; Hammond \& Adelman,
1976). In doing so, he drew attention to the dichotomy between theories
of correspondence, which focus on empirical reality, and theories of
coherence, which focus on internal consistency, as well as the issues
of separating facts from values and balancing type I and II errors in
judgment under uncertainty. In the remainder of this paper, we discuss
each of Hammond’s major contributions in more detail.

\section{Keeping Egon Brunswik’s Legacy Alive}

Hammond was key to keeping alive and advancing the legacy of the
Austro-Hungarian psychologist Egon Brunswik who worked largely in the
field of perception (1940, 1943, 1944, 1952, 1955c, 1955a, 1956; Tolman
\& Brunswik, 1935; for a review of Brunswik’s ideas, see Dhami,
Hertwig \& Hoffrage, 2004). Specifically, Hammond (1966) edited
\textit{The Psychology of Egon Brunswik}, which appeared just over a
decade after Brunswik’s untimely death. This volume contained a eulogy
by Edward Tolman; essays on Brunswik’s contributions to the science of
psychology by such luminaries as Donald Campbell, Jane Loevinger, Fritz
Heider, Lee Cronbach, and Roger Barker; four key papers by Brunswik
himself; and a bibliography of Brunswik’s published works. Second,
Hammond co-edited \textit{The Essential Brunswik }(Hammond \& Stewart,
2001b), which reprinted eighteen of Brunswik’s major papers accompanied
by original commentaries, along with over two dozen papers discussing
Brunswik’s theoretical and methodological contributions to psychology
and describing applications of Brunswikian psychology to substantive
problems. This volume also provided a complete annotated list of
Brunswik’s publications. Third, Hammond established The Brunswik
Society, with its associated annual international meeting and
newsletter. This provided a setting for intellectual exchange among the
next generation of neo-Brunswikian scholars.

In addition to his concrete efforts in archiving, preserving and
curating Brunswik’s legacy, Hammond also built on Brunswik’s work. He
was not deterred by the fact that Brunswik’s ideas were resisted and
ignored by his contemporaries. Rather, Hammond vigorously championed
Brunswik’s theory of probabilistic functionalism and lens model, as
well as his method of representative design. 

Brunswik (1943, 1952) believed that an organism functions to achieve a
distal variable in its environment. This is done by using multiple,
proximal cues that may be inter-correlated and are ultimately fallible.
He argued that psychological processes are adapted to the probabilistic
environments in which an organism functions. Brunswik (1957) saw the
organism and environment as equal partners. Hammond and Stewart (2001a)
suggest that the clearest statement of this fundamental thesis can be
found in Brunswik’s (1957, p.\ 5) last paper, where he wrote:
\begin{quote}
….both organism and environment will have to be seen as systems, each
with properties of their own, yet both hewn from basically the same
block. Each has surface and depth, or overt and covert regions….the
interrelationship between the two systems has the essential
characteristic of a “coming-to-terms.” And this coming-to-terms is not
merely a matter of mutual boundary or surface areas. It concerns
equally as much, or perhaps more, the rapport between the central,
covert layers of the two systems. It follow that, much as psychology
must be concerned with the texture of the organism or of its nervous
processes and must investigate them in depth, it must also be concerned
with texture of the environment as it extends in depth away from the
common boundary.
\end{quote}

The theory of probabilistic functionalism had analytic and
methodological corollaries. Brunswik (1944, 1955c, 1956) argued that in
order to understand how the organism has adapted to its environment,
researchers should employ the method of representative design ideally
in the form of randomly sampling stimuli from a defined population to
which the researcher wants to generalize. This contrasts from the
commonly used systematic design (i.e., manipulation and control of
selected variables) in psychology. In addition, Brunswik (1943) noted
that in order to measure an individual’s degree of achievement, the
researcher needs to perform data analysis at the idiographic level.
This departs from nomothetic analysis, which is most commonly used in
the law finding tradition. 

Hammond (1966, p.\ 16) characterized Brunswik’s theory of probabilistic
functionalism as “an organic whole — a history, comprehensive theory, and
a methodology.” He took the lead in extending probabilistic function so
that it applied to judgment and decision making processes as well as to
perception, which was Brunswik’s original and primary focus. In the
last few years of his life, Brunswik introduced the concepts of
\textit{quasirationality} (Brunswik, 1952) and \textit{ratiomorphic}
models of perception and thinking (Brunswik, 1955b; 1956, pp.\ 89--93).
These nascent efforts were also advanced by Hammond, and four decades
after Brunswik’s death, Hammond (2001a) chronicled his own success in
applying the lens model to clinical judgment, conducting research on
learning under uncertainty, developing a mathematical representation of
the lens model, expanding this model to studying interpersonal learning
and conflict, and developing cognitive continuum theory.\footnote{
Hammond continued to perform idiographic analysis (but see Hammond et
al., 1975), as Brunswik suggested, despite its time-consuming and
somewhat tedious nature. } We will discuss these contributions in more
detail in subsequent sections of the paper. But, first, we will
consider Hammond’s efforts to translate Brunswik’s method of
representative design and his frustrated struggle to advocate
Brunswik’s ideals for a scientific psychology. 

As Hammond (2001b, 2001c) lamented, representative design is not
well-understood in psychology (see also Dhami, 2011; Dhami et al.,
2004; Gigerenzer, 2001). Hammond (2001b; p.\ 134) pointed out that it
does not refer to: 
\begin{quote}
\dots a demand for representation of the “real world” (a meaningless concept)
but…to the extent to which the statistical properties of the laboratory
task represent the statistical properties of the situation to which the
results are to be generalized. In short, the same logic that is used to
justify generalization over a subject population should be used to
justify generalization over the task situation.
\end{quote}

Hammond was tenacious in his promotion of representative design. It was
a key element of his first paper (Hammond, 1948), a key element of his
last (posthumous) paper (Hammond \& Lang, 2017) and many papers in
between (e.g., Hammond, 1996b). In particular, he clarified and further
developed Brunswik’s ideas regarding representative design by
distinguishing between substantive situational sampling and formal
situational sampling (Hammond, 1966, 1972).

Formal situational sampling refers to the formal properties of the task
(i.e., number of cues, their values, distributions, inter-correlations,
and ecological validities), irrespective of content. The formal
properties define the universe of stimulus (or situation) populations.
For instance, cue number, values, and distributions range from 0 to
infinity, and the ecological validities of cues and their
inter-correlations range from -1 to 1. Any population of situations
lies within these boundaries. A researcher who uses formal situational
sampling can sample various combinations of formal properties. Formal
situational sampling permits the construction and presentation of
stimuli that are formally representative of the natural stimulus
population. In order to discover what the formal properties of the task
are, it is often necessary to conduct a task analysis prior to the
study. This may involve interviews with those who are familiar or
experienced with the task, observations of individuals performing the
task, document analyses of past case records and a review of the extant
literature on the task.

% Figure 1
\begin{figure*}[t]\centering
\includegraphics[width=4.5in]{Figure1.eps}
\caption{The Lens Model (adapted from Brunswik, 1952).}
\end{figure*}

An example of the use of formal situational sampling can be found in
Hammond et al.’s (1987) study of 21 expert highway engineers judging
highway safety. The distal criterion was the rate of accidents divided
by the number of miles travelled, averaged over 7 years, for each of 40
highways. The highways were described in terms of 10 cues (e.g., lane
width) that highway safety experts identified as essential for judging
road safety. The values, inter-correlations, distributions, and
ecological validities of the eight cues were deduced from highway
department records, and properties of two cues were measured by the
experimenters from visual inspection of videotapes of each highway. The
researchers were primarily interested in examining how presentation of
the task affects the mode of cognition used (e.g., analytical versus
intuitive), and so the cue information for each highway was presented
via filmstrips, bar graphs, and formulas.

Despite Hammond’s proposal to use formal situational sampling, as
Brehmer (1979, p.\ 198) argued, there is “no easy road to success”. The
number of all possible combinations of variables may be extremely
large, and the researcher needs to know which combinations should be
studied. The problem of defining a reference class or sampling frame
remains. In a review, Dhami et al.\ (2004) revealed that when trying to
capture judgment processes, few researchers had used representative
design either in terms of sampling real stimuli from the environment
(for some exceptions, see Kirlic, 2006) or
in terms of constructing stimuli to be formally representative of the
environment. Hammond (1996b, p.\ 245) himself confessed that he had not
always been faithful to the method of representative design and
explained that this was because, like Brunswik, if he had done so, “I
would become isolated and ostracized.”\footnote{ Hammond was thanked by
researchers who later drew on Brunswik’s idea of representative
stimulus sampling to critique the extant work on cognitive biases such
as overconfidence (e.g., Gigerenzer, Hoffrage \& Kleinbolting, 1991;
see also Juslin, Winman \& Olsson, 2000). Simply put, it was found
that biases were reduced if researchers employed representative rather
than systematic stimulus sampling.} 

\section{Clinical Judgment, Policy Capturing and Judgment Analysis}

Hammond (1955) was the first to use the lens model outside the study of
perception. He pioneered the use of Brunswik’s (1952) lens model (see
Figure 1) in the study of clinical judgment, employing it as a
framework for studying how individuals use information from the task
environment to make clinical judgments. This work was the precursor to
‘policy capturing’/‘judgment analysis’ research mentioned earlier.

In his 1955 landmark paper, Hammond shifted attention away from the
practice of solely studying the accuracy of clinical judgment to also
explaining how clinicians achieve accuracy.\footnote{At about the same
  time, Meehl (1954) published an influential book comparing clinical
  versus actuarial prediction, concluding that statistical predictions
  typically outperformed clinical judgment.} This was a fundamental
shift in focus because “it is suggested that the clinician not be
considered a reader of instruments, but an instrument to be understood
in terms of a probability model” (Hammond, 1955, p.\ 262). Hammond
argued that the clinician and the patient are two different, but
interacting systems that should be considered as a whole, and so
research should examine the relations between a clinician and his or
her environment (i.e., patients). He further pointed out that the
clinician’s judgment process is often ‘quasi-rational’ and difficult
to communicate because it is a result of the process of vicarious
functioning. The term quasi-rational was first used by Brunswik (1952)
to describe cognitive processes that were intermediate between
intuitive and analytical poles, and Hammond elaborated this concept in
other work (Hammond, 1996a; Hammond et al., 1987), which we will
return to later.

Brunswik (1956, 1957) used the concept of vicarious functioning to
describe how, in an uncertain environment in which no proximal cue is a
perfectly valid and reliable indicator of the distal state, organisms
learn to rely on multiple cues of partial but imperfect validity and
reliability. Vicarious functioning is essential in clinical judgment
because clients may present a set of symptoms that may change over time
or may present symptoms different from those presented by another
client who is suffering the same problem. Hammond also argued that a
clinician’s capacity for dealing with the intersubstitutability of
cues, over encounters with a series of patients, should be studied
using representative design.

Hammond’s points are illustrated in two studies conducted by his
students (Herring, 1954; Todd, 1954, see also Hammond, 1955). In the
first study, Todd (1954) asked 10 clinicians to judge the intelligence
(as measured by an IQ test) of 78 patients using a Rorschach test. The
IQ test score was the objective outcome criterion. Achievement was
measured in terms of the correlation between the clinician’s judgments
made over a set of clients and their IQ test scores. Emulating
Brunswik’s (1940) approach, correlational statistics were used to
capture the relationships between cues and judgments (cue utilization),
cues and the environmental criterion (ecological validity), as well as
inter-cue correlations. 

Hammond noted that the clinicians’ performance improved when they had
access to more information. The median correlation between judgments
and IQ scores rose from 0.47 to 0.64 when clinicians had access to the
verbal protocol data from the Rorschach. Then, using the four most
valid cues as predictors, a multiple linear regression equation was
computed for the environment (i.e., capturing the relations between the
patients’ IQ scores and the four Rorschach cues) and separate equations
were computed for each clinician (i.e., capturing the relations between
each clinician’s judgments of IQ and the four Rorschach cues). Each
model revealed the relative weights attached to the cues. The model of
the environment was then compared to each clinician’s model.\footnote{
Whereas Meehl (1954) had compared the clinician to a statistical
model of the environment, Hammond (1955), compared a statistical model
of the clinician to a statistical model of the environment.} The
match between the two models explained how the clinicians attained
their level of achievement. The multiple $R$ for the model of
the environment was 0.479 and the median correlation between the
clinicians’ judgments of IQ and the IQ test score was 0.470. The
ecological validities of the four cues in the environment, each
clinician’s utilization validities of the four cues, the
inter-correlations among the four cues in the environment and as used
by each clinician, were also elicited. There were variations among the
clinicians in terms of the cues they used. “Certain clinicians were
found to be using invalid cues, others neglected the valid ones”
(Hammond, 1955, p.\ 261). 

In his study of clinical judgment, Hammond also demonstrated that the
multiple linear regression model could predict human (clinical)
judgment. A model was developed for each clinician on 39 patients and
cross-validated on a further 39 patients (i.e., it made predictions of
each clinician’s judgments on a set of new cases). The median
correlation between the models’ predictions and the clinicians’
judgments on the new cases was 0.85. Hammond (1955, p.\ 261) concluded
that “evidently the multiple correlation model which predicts that the
clinician combines the data from the Rorschach in a linear, additive
fashion is a good one – it predicts quite successfully in comparison with
most psychological efforts.” 

In the second study, Herring (1954) asked clinicians to judge patients’
responses to surgical anesthesia using their psychological test
results. No objective outcome criterion was available and so analysis
of the environment side of the lens model and consequently achievement
was not possible. Instead, Hammond demonstrated how the correspondence
or agreement between the judgments of two clinicians (a medic and a
psychologist) could be studied by examining the match between their
regression models. Brunswik (1956, p.\ 30) had also noted that
correlations could be used to measure “agreement among judges.”

In his first successful application of the regression model to clinical
judgment, Hammond (1955, p.\ 261) was cautious in noting that the
analyses merely demonstrated that it was possible to construct
“\textit{some} probability model” that captured the essential elements
of the lens model. Over time, however, the similarity between the
multiple linear regression model of the environment and of the
clinician’s judgments led Hammond et al.\ (1964, p.\ 444) to conclude
that “The clinicians’ inferential processes were nearly identical with
the multiple-regression procedure both in \textit{function} and in
\textit{content.}” When proposing further studies, they stated that “We
are confident that … such studies will find small differences between the
cognitive processes of the clinician, or any human subject, and the
multiple-regression equation” (p.\ 452). Later, however, Hammond (1996b,
pp.\ 244--245) confessed that: “a […] sin of commission on my part was to
overemphasise the role of the multiple regression (MR) technique as a
model for organising information from multiple fallible indicators into
a judgment. There is nothing within the framework of the Lens Model
that demands that MR be the one and only model of that organising
process.” Researchers have now successfully developed and tested
non-statistical alternatives to the regression model within the lens
model context (e.g., Dhami \& Ayton, 2001; Dhami \& Harries, 2001;
Gigerenzer \& Goldstein, 1996).

Hammond’s ground-breaking 1955 paper presaged two related, but
distinguishable threads of work. In one thread, exemplified by the
Todd study, the focus is on modelling both the human judgment and
environment side of the lens model, as well as the interaction between
them, consistent with the fundamental tenets of Brunswik’s
probabilistic functionalism. In the other thread, reflected by the
Herring study, the emphasis on the environmental system recedes
(because it is unknowable, unavailable or not of interest), and the
spotlight is on how individual judges weight and combine information
to make judgments. This latter situation represents what has been
called the single-system design (Cooksey, 1996; Hammond, 2001a) of the
lens model (see Figure 2) and is associated with the
policy-capturing/judgment-analysis approach (Brehmer \& Brehmer, 1988;
Stewart, 1988), which we discuss in more detail below.

In the single-systems design there is no outcome criterion, or at least
none that is immediately available, so researchers simply describe an
individual’s judgment policy. As Cooksey (1996) points out, the
single-system design is not truly Brunswikian because when the
environment is unknown or unmeasurable, it does not allow for an
examination of the interrelationships between the judge’s cognitive
system and the task system. Nonetheless, Hammond (2001a) noted that the
single-system design is the most widely used version of the lens model.
Dhami et al.\ (2004) also observed that much of the subsequent research
within SJT, developed by Hammond and his colleagues, which we discuss
later, uses this design. 

% Figure 2
\begin{figure}[t]\centering
\includegraphics[width=2in]{Figure2.eps}
\caption{Lens Model for single-systems design.}
\end{figure}

The single-system design may be appropriate because an outcome criterion
is unavailable, for a variety of valid reasons, as enumerated by
Cooksey (1996). First, an outcome criterion may not be useful or
appropriate because no definitive, correct answer is available or
ascertainable. In order to overcome problems in obtaining an outcome
criterion, some studies have used expert judgments to provide proxies
for environmental criterion measures (e.g., Adelman \& Mumpower, 1979;
Hammond \& Adelman, 1976; Mumpower \& Adelman, 1980). Second, an
outcome criterion may be difficult to obtain due to concerns with
ethics or legality. Third, an outcome criterion may be unavailable
during the study period. Fourth, studies using hypothetical cases or
cases that represent future situations, by their very nature preclude
the use of an outcome criterion. Finally, an outcome criterion may not
be included because it is irrelevant to the research goal since
researchers wish solely to study agreement.

As mentioned above, analyses of how judges weight and combine
information to make judgments in the single-system design have come to
be known to as policy capturing and judgment analysis both within and
outside\footnote{There is a large body of basic and applied
  psychological research that arose independently, and has little or
  no apparent connection to Brunswik or Hammond. Some have argued that
  the history of policy capturing research dates back to Wallace
  (1923) who is often credited with pioneering such efforts in his
  study of corn judges in which he tried to capture ‘what is in the
  corn judge’s mind?’  The term policy capturing was coined by
  Bottenberg and Christal (1961), and refers to the analysis of
  judgment data using multiple regression techniques. Judgment
  analysis (originally called JAN) was coined by Christal (1963), and
  refers to the combined use of policy capturing and policy clustering
  methods. } the Brunswikian and SJT traditions. A large body of
research has emerged, and in Brehmer and Joyce’s (1988) edited volume
called \textit{Human Judgment: The SJT View} that summarizes advances
since Hammond et al.’s (1975) introduction of SJT, there is both a
chapter on judgment analysis (Stewart, 1988) and a chapter on policy
capturing (Brehmer \& Brehmer, 1988). The former chapter acknowledges
that the topic of that chapter is sometimes also called ‘policy
capturing’ and the latter chapter similarly acknowledges that the
focus of the chapter is sometimes referred to as ‘judgment
analysis’.\footnote{ Cooksey (1996, pp.\ 57--58) provides an
  informative discussion of the points of intersection and divergence
  between the strands of policy capturing and judgment analysis
  research closely linked to the Brunswikian and SJT tradition and
  those that are outside that tradition.}

Judgment analysis and policy capturing have been fully explicated by a
number of scholars (Brehmer \& Brehmer, 1988; Cooksey, 1996; Hammond
et al., 1975; Stewart, 1988). A study by Dhami and Ayton (2001)
provides an example of the primary features of this approach. They
studied British judges’ bail decisions. Through a postal survey, 81
judges were presented with a set of 41 hypothetical cases. These were
described in terms of nine cues (e.g., seriousness of offence,
community ties). The cues were identified following a review of the
literature on bail, legal guidelines and training, interviews with
judges, and observations of bail hearings. Each cue itself varied
(e.g., the strength of community ties cue was dichotomous and
described as strong or weak). Other cues (e.g., length of adjournment)
were held constant and provided background information to all the
cases. The judges were first asked to make a decision on each case
(i.e., bail unconditionally, bail with conditions, and remand in
custody). They were then asked to rate how confident they were in
their decision.  After making their decisions, the judges were asked
to rank the cues in order of the importance attached to them when
making their decisions.  Seven of the 41 cases were duplicates used to
measure test-retest consistency in decisions, and seven were holdouts
used for model validation. Finally, studies interested in achievement
would also document the outcome (or criterion) for each case, although
this was not possible for Dhami and Ayton (2001).

Once the judgment data has been collected, judgment policies are
captured for each individual.\footnote{ In some studies, aggregate or
composite policies may be captured (Hammond et al., 1975), although
this departs from Brunswik’s (1943) call for analysis to be at the
idiographic level. } Traditionally, policies are captured using
multiple linear regression statistics (Cooksey, 1996; Hammond et al.,
1975; Stewart, 1988). The criterion (dependent) variable is the
judgment and the cues are the predictor (independent) variables. An
individual’s judgments are regressed on the cues. This procedure yields
a weighted linear model that can be cross-validated, and which
describes an individual’s judgment policy in terms of statistically
significant cues in the model, relative cue weights, the form of the
function relating the cues to the judgments (e.g., linear), the rule
used to integrate the cues into a judgment (i.e., additive), and an
individual’s predictability as measured by the model (i.e.,
$R_2$). Inter-individual differences (or
agreement) in decisions and policies may then be examined.
Participants’ ‘insight’ into their decision making may also be examined
by comparing their self-reported policies with their model. Finally,
intra-individual inconsistency in making decisions may be studied by
comparing the decisions made in the test-retest situations.

Although there are exceptions, as Brehmer and Brehmer (1988) point
out, reviews of judgment analysis and policy capturing research have
generally concluded that the studies have yielded consistent findings,
irrespective of the number and type of decision makers sampled, and
the nature and content of the judgment tasks involved (Brehmer, 1994;
Brehmer \& Brehmer, 1988; Camerer, 1981; Cooksey, 1996; Libby \& Lewis, 1982; Hammond
et al., 1975; Karelaia \& Hogarth, 2008; Slovic \& Lichtenstein,
1971).\footnote{ Slovic and Lichtenstein (1971) reviewed studies
  conducted in what they called the correlational and analysis of
  variance paradigms. Hammond et al.\ (1975) reviewed experimental
  studies in the multiple-cue probability learning, and the
  interpersonal learning and interpersonal conflict paradigms. Libby
  and Lewis (1982) reviewed studies conducted in the domains of
  accounting and auditing.  The reviews by Brehmer and Brehmer (1988)
  and Brehmer (1994) focused on studies involving participants
  experienced with the judgment task being studied. Finally, Cooksey
  (1996) reviewed studies employing the single-systems,
  double-systems, triple-systems, N-systems and hierarchical designs
  (we shall say more about these designs later).}  Studies have
typically found that the multiple linear regression model is a
descriptively valid model as it provides an adequate fit to
individuals’ judgment data. Researchers have suggested that a high
$R_2$ implies that judgments are the result of
a linear additive process (Hammond et al., 1964. However, as we
pointed out earlier, Hammond recognized the paramorphic nature of the
regression model, and other researchers have since successfully
captured human judgment policies using non-statistical and even
non-compensatory models (e.g., Dhami \& Ayton, 2001; Dhami \& Harries,
2001, 2010). Gerd Gigerenzer and his colleagues initiated a program of
research modeling the task environment, in which they demonstrate the
descriptive and predictive utility of simple (non-statistical and
sometimes non-compensatory) models, called ‘fast and frugal’
heuristics (e.g., Gigerenzer, Todd \& the ABC Research Group, 1999;
see also Gigerenzer, Hertwig \& Pachur, 2011).

% not sure Camerer reference goes here

The regression model indicates the number of cues used by an individual
(i.e., those that have statistically significant beta weights), the
relative weights of the cues (i.e., beta weights) and the direction in
which cues were used (i.e., the sign of the beta weight). It has
typically been found that regression models contain on average three
cues (Brehmer, 1994; Slovic \& Lichtenstein, 1971). Most studies have
reported that participants do not use all of the cues available.
Studies have often found that a few cues are weighted more heavily than
the others. These findings on cue use are compatible with the more
recent research demonstrating the descriptive validity of
non-compensatory models that imply judgments may be based on only one cue
(e.g., Dhami \& Ayton, 2001; Dhami \& Harries, 2001; Gigerenzer \&
Goldstein, 1996).

Hammond et al.\ (1975) proposed that consistency should be measured in
terms of the variance of judgments made in a test-retest situation.
Studies using such measures of consistency have often correlated the
two sets of decisions, although other indices of agreement may also be
used (e.g., Gillis, Lipkin \& Moran, 1981). It has generally been
found that correlations are moderate for the majority of participants
in a study. Note that inconsistency implies some degree of judgment
inaccuracy. 

Researchers have also compared the captured policies with individuals’
stated policies, as elicited by various direct report
methods.\footnote{ The policy captured by the regression model is also
referred to as a tacit, implicit or objective policy, and the
individual’s own statement of policy is also called his or her explicit
or subjective policy. Cook and Stewart (1975) found little difference
amongst seven different direct report procedures, namely distributing
100 points among cues, rating cues on a 100 point scale, paired
comparison ratings of cues, ratio comparison ratings of cues, the
number of times cues were influential, aggregation of judgments made
using each cue one at a time on each case, and aggregation of ratings
of each cue’s contribution to the judgment of each case. } The
subjective cue weights may be compared with statistical weights derived
from the regression model, the fit of models containing each set of
weights may be compared, or the predictions made by the two sets of
weights may be compared (Reilly \& Doherty, 1992). According to some,
such comparisons provide a measure of an individual’s insight into his
or her judgment policy (e.g., Ullman \& Doherty, 1984). Todd (1954; see
also Hammond, 1955 and Summers, Taliaferro \& Fletcher, 1970) had
found that his participants were not able to accurately articulate the
weights they attached to the cues. However, acknowledging that direct
methods may provide an unreliable and invalid method for demonstrating
self-insight because of the difficulties in introspection and
articulating policies, Reilly and Doherty (1989, 1992) proposed an
alternative method, i.e., policy recognition. Here, participants are
asked to identify their own policy, defined in terms of cue weights,
from a set of other policies. This method indicates a greater degree of
insight, as participants were quite successful in recognizing their own
policies.

\section{The Lens Model Equation}

Hammond and his colleagues introduced the lens model equation (Hammond
et al., 1964; Hursch et al., 1964) to the study of judgment processes.
This equation provided the requisite quantitative tool for modeling and
analyzing both the environment side and the human judgment side of
Brunswik’s lens model. The lens model equation is the analytic
complement of the conceptual framework articulated in the lens model.
The original formulation by Hursch et al.\ (1964) was simplified by
Tucker (1964), whose version is shown in Equation 1 (see also Karelaia
\& Hogarth, 2008).
\begin{equation}
r_a = G R_e R_s + C \sqrt{1-R_e^2} \sqrt{1-R_s^2}
\end{equation}
Here, $r_a$ represents achievement, and is
measured by the correlation between the judgments and the criterion. 

$R_e$ represents the predictability of the
environment and thus the upper limit of achievement. It is measured by
the linear multiple correlation between the cues and the criterion.

$R_s$ represents an individual’s ability to
utilize his or her knowledge of the task in a consistent manner, and is
measured by the linear multiple correlation between the cues and the
judgments.

$G$ represents the match of the linear components of the two
models, namely the model of the environment and of the individual. It
is measured by the correlation between the linearly predictable
variance in the environment and the linearly predictable variance in
the individual’s judgments. 

$C$ represents the non-linear component of achievement, and is
measured by the correlation between the residuals from the linear
regressions of the environment and the individual.

Each of the major statistical terms of the lens model equation can be
directly translated into the conceptual framework of Brunswik’s lens
model itself. \textit{Achievement} ($r_a$) is determined by an
individual’s ability to detect and utilize both linear and non-linear
patterns in the environment. Stewart (1988) notes that in most
judgment tasks, the non-linear component is so small as to be
negligible. If the non-linear component is large, it indicates that
the multiple linear regression model is not capturing all of the
consistent variation in judgment.

\textit{Task predictability} ($R_e$) is measured in terms of the
linear multiple correlation between the cues and the
criterion. Perfect achievement is not possible when $R_e$ is less than
1. It is not possible to be always correct in a world that is not
perfectly consistent. Task predictability thus sets an upper
mathematical bound on achievement; $r_a \le R_e$. Brehmer (1970, 1972,
1973b) demonstrated that not only does $R_e$ set an upper bound on
$r_a$, but that lower levels of task predictability tend to induce
sub-optimal levels of achievement. This happens because less
predictable tasks tend to elicit lower levels of cognitive control,
$R_s$. Brehmer (1976) suggests that judges become less consistent as
they modify their judgment policies in order to try to achieve higher
levels of achievement, but due to task uncertainty they receive noisy
feedback, making it difficult to assess whether changes in judgment
policy have led to improved performance. As summarized by Stewart,
Roebber, and Bosart (1997, p.\ 206), “Not only does task
predictability place an upper bound on potential accuracy \dots, but a
number of studies have found evidence that the reliability of judgment
is lower for less predictable tasks \dots Judges respond to
unpredictable tasks by behaving less predictably themselves.”

\textit{Cognitive control} ($R_s$) refers to the relationship between
an individual’s judgments and environmental cues.  It is measured in
terms of the linear multiple correlation between cues and
judgments. Building on earlier work, Hammond and Summers (1972)
proposed that performance in cognitive tasks involves two distinct
processes: acquisition of knowledge and cognitive control over
knowledge already acquired. Just as task predictability places an
upper bound on achievement, so does cognitive control. Cognitive
control sets an upper mathematical bound on achievement;
$r_a \le R_s$. In subsequent work, Hammond and colleagues (Hammond et
al., 1975) distinguished between cognitive control and
consistency. Cognitive control refers to the similarity between an
individual’s judgments and predictions of the best-fitting model of
that individual. Consistency refers to the similarity between repeated
judgments made on identical cases. Consistency places an upper bound
on cognitive control.

\textit{Knowledge} ($G$) is assessed in terms of the correlation
between the best fitting model of the human system and the best fitting
model of the environmental system. It represents the achievement that
would be attained if both the model of the judge and the task were
executed with perfect control. In general, $G$ will be maximized
when the judge’s cue utilizations (weights and function forms) parallel
the ecological validities of the environmental cues, presuming that the
judge uses the same organizing principle to combine the information
into an overall judgment as is found in the environmental system. As
Karelaia and Hogarth (2008) note, however, it is possible for judges to
obtain high levels of $G$, even when the cue utilizations do not
precisely match the ecological validities. This can happen when
inter-cue correlations are high. 

Finally, $C$ is also a measure of knowledge. It represents the
consistent non-linear variation shared between the individual and
environment that is not captured by the regression models.

Cooksey (1996, p.\ 165, words in square brackets added) sums up the
situation nicely: 
\begin{quote}
The LME [lens model equation] is an elegant, precise mathematical
formulation of a simple truth. That is, a person’s ability to make
correct judgments about reality is a function of three things: (1) how
predictable the world is ($R_e$), (2) how well
the person knows the world ($G$ and $C$), and (3) how
consistently the person can apply his or her knowledge
($R_s$).
\end{quote}

\renewcommand{\baselinestretch}{1.1} \large\normalsize

In the first application of the lens model equation, Hammond et al.
(1964) reanalyzed data from a study by Grebstein (1963), who compared
naïve, semi-sophisticated and sophisticated clinicians’ predictions of
30 patients’ IQ test scores using 10 cues from patients’ Rorschach
tests. Grebstein had concluded that performance did not increase with
experience and that there was room for improvement. Hammond et al.\
(1964) used the lens model equation to determine the upper limit of
achievement for this task and found that $R_e$ was 0.79. They also
demonstrated that the three groups of clinicians did not differ in
terms of $R_s$, or $C$ as all groups were highly linear. The three
groups did, however, differ in terms of $G$ because clinicians in the
more sophisticated groups exhibited a greater match between the
ecological validities of the cues and their utilization validities.

Once individuals’ judgment policies have been captured, they can be
compared with the model of the task, in order to examine individual
achievement. As mentioned, the upper limit of achievement is dependent
upon that afforded by the task. Libby and Lewis (1982) concluded that
studies in accounting (in particular prediction of business failure
and prediction of security return) reported high levels of
achievement.  Overall, studies comparing judgments against an outcome
criterion have demonstrated fairly high levels of achievement or
judgmental accuracy as measured by the lens model equation; $r_a$
averaged 0.56 among the 249 studies in a meta-analytic review by
Karelaia and Hogarth (2008).

Since the first studies were conducted over 50 years ago, an extensive
body of research has used or referred to the lens model equation. The
precise number of studies is difficult to estimate accurately, but
efforts to compile bibliographies (Holzworth, 1999) and to conduct
meta-analyses (Karelaia \& Hogarth, 2008) provide evidence that there
are hundreds of such studies, at a minimum. Methodologically, Castellan
(1973, 1992), Cooksey and Freebody (1985), Stewart (1976, 1988, 1990)
and Stewart and Lusk (1994), among others, have made important
modifications, suggested significant extensions, or examined the
statistical and empirical behavior of parameters of the lens model
equation. 

By extending Brunswik’s lens model from the study of perception to
judgment, and then formulating the lens model equation, Hammond opened
up the possibility of new research approaches and agendas. These tools
could be used to study how individuals learn to make judgments in
probabilistic multiple-cue learning situations. We discuss this topic
in the next session. Second, these tools offered the opportunity to
study inter-individual (dis)agreement in judgment, interpersonal
learning, and interpersonal conflict. This topic is discussed in the
subsequent section.

\renewcommand{\baselinestretch}{1.00} \large\normalsize

\section{Multiple-Cue Probability Learning}

The lens model equation enabled Hammond to undertake a program of
research on multiple-cue probability learning in which he and others
made numerous contributions. Holzworth (1999) provides a good overview
of these efforts:
\begin{quote}
Cue probability learning involves an organism attempting to achieve
(learn) a relationship with some distal criterion variable by attending
to one or more multiple fallible indicators (differentially valid
cues). Smedslund (1955) conducted the first multiple and single cue
probability learning study after Brunswik (Brunswik \& Herma, 1951),
but it was Hammond and his students in the United States (Hammond,
Hursch \& Todd, 1964; Hursch, Hammond \& Hursch, 1964), and Björkman
(1965) and his student Brehmer (1972) in Sweden who initiated extensive
programs of research. During a typical cue probability learning
experiment a person makes judgments based on some number of
probabilistic cues over a series of trials. The object is to correctly
predict the quantitative or categorical criterion value on each trial.
Cues differ in terms of their relevance (ecological validity) to the
criterion. Trial by trial (outcome) feedback may be given on each
trial, and/or cognitive feedback may be given after subsets of trials.
Cognitive feedback concerns characteristics of the person’s cognitive
processes as well as characteristics of the task ecology.
\end{quote}

In one of the earliest multiple-cue probability learning studies,
Hammond and Summers (1965) gave three groups different amounts of
information about the task in addition to different amounts/forms of
outcome feedback (i.e., no information, information that the task
contained linear and non-linear cue-criterion function forms, and
information that in addition identified the linear and non-linear
cues). They asked individuals to predict a criterion value from two
cues, one linearly related to the criterion and one non-linearly
related. All groups showed learning over five blocks of 20 trials and
all groups learned to use the linear cue more efficiently than the
non-linear cue. However, individuals in the group given the most
information showed a greater degree of achievement and were more likely
to learn to use the non-linear cue. The two groups who were provided
information about the task were also better at learning the ecological
validities of the cues. 

Holzworth’s 1999 annotated bibliography identified 315 studies of
single- and multiple-cue probability learning conducted up to that
date. Several important empirical regularities have emerged from the
literature. For instance, people can learn positive cue-criterion
relations faster than negative ones; they can slowly learn to track
changes in relative cue weights over time; they can learn to use cues
faster than learning function forms; and they typically do not use cue
redundancies effectively (Klayman, 1988; Slovic \& Lichtenstein,
1971).  Generally speaking, the findings from the multiple-cue
probability learning paradigm indicate that individuals can learn
about the formal properties of the task such as cue validities and
linearity of cues and adapt to them, although learning tends to be
better and faster when task predictability ($R_e$) is higher (Brehmer,
1973a; 1974). It has also been shown that people find it more
difficult to learn non-linear cue-criterion relations than linear ones
and they experience difficulty in applying knowledge about non-linear
relations consistently (Deane, Hammond \& Summers, 1972).

Hammond initiated a line of research within the multiple-cue
probability learning paradigm that proved to be of particular
consequence. This concerned the distinction between, and differential
effects of, outcome versus cognitive feedback. It was clear from
Hammond and Summers’ (1965) study that providing information about
properties of the task in addition to traditional outcome feedback
improved learning. Thus, Todd and Hammond (1965) developed the
procedure of cognitive feedback (sometimes also known as lens model
feedback).\footnote{ Cognitive feedback is the term used when
  information about past events is provided and cognitive feedforward
  refers to information about future events (Doherty \& Balzer,
  1988).} This involves providing information about the formal
properties of the task, the individual’s judgment policy (i.e.,
utilization validities, $R_s$, consistency and cue-judgment function
forms), and the match between properties of the environment and the
individual’s judgment policy (i.e., achievement, $G$ and $C$; Balzer,
Doherty \& O’Connor, 1989; Doherty \& Balzer, 1988).

Todd and Hammond (1965) provided participants with feedback on their
level of achievement, cue utilization validities and the ecological
validities of the cues, for each of eight blocks of 25 trials. They
found that cognitive feedback led to significantly greater achievement
than did outcome feedback,\footnote{Newton (1965) had
  demonstrated that the sole provision of cognitive feedback (i.e.,
  ecological validities of cues and the utilization validities) could
  significantly improve performance.} and concluded that cognitive
feedback enables people to compare their understanding of the task and
discover where they were using cues inappropriately. This study was
followed by a series of additional ones which revealed that
participants given only outcome feedback learned less rapidly than
those given cognitive feedback (Deane et al., 1972; Hammond, 1971;
Hammond \& Boyle, 1971; Hammond, Summers \& Deane, 1973).

Other research has found that providing both cognitive feedback in the
context of outcome feedback may actually impair performance (Holzworth
\& Doherty, 1976); that learning is slow and difficult with outcome
feedback alone (Brehmer, 1980; Klayman, 1988);\footnote{ Unfortunately,
this latter situation reflects how learning may take place outside the
laboratory (Anderson, Deane, Hammond, McClelland \& Shanteau, 1981).}
and that cognitive feedback is superior to no feedback at all (Doherty
\& Balzer, 1988). Unlike cognitive feedback, in stable environments,
outcome feedback does not provide information useful for making future
judgments. 

% Figure 3
\begin{figure*}[t]\centering
\includegraphics[width=4.5in]{Figure3.eps}
\caption{Lens model for study of interpersonal learning and
interpersonal conflict (adapted from Hammond [1965] and Hammond et al.
[1966b]).}\vspace{-2ex}
\end{figure*}

The usefulness of cognitive feedback, however, may need to be buttressed
by other information. Balzer et al.\ (1989) concluded that providing
people with information about the characteristics of their own judgment
performance but not about the appropriateness of that performance does
little to improve performance compared to providing information about
the characteristics of the task. More recently, Karelaia and Hogarth
(2008) concluded that information about the task was more useful than
feedback about the judgment policy alone, or simply outcome feedback.
As Brehmer (1979) points out, in real world conditions feedback is not
always available and people are not consciously trying to learn the
task. In fact, analyzing feedback requires cognitive resources, which
may be lacking in complex tasks (Harvey, 2011). 

\section{Interpersonal Learning and Interpersonal Conflict}

Among Hammond’s most significant contributions was his extension of the
lens model to social contexts in which individuals learn from or about
one another, or are in judgment conflict with one another. Brunswik’s
probabilistic functionalism is based on the double-system lens case,
where the two focal systems are the individual and the environment.
This asocial focus is unsurprising given that Brunswik was primarily
concerned with perception. For Brunswik (1956, p.\ 35), the study of
agreement was the only time when more than one individual was “brought
into the picture.” 

In an innovative and revolutionary development, Hammond moved the lens
model into a social context, introducing both the interpersonal
learning (IPL) paradigm (Hammond, 1972; Hammond et al., 1966a; see also
Earle, 1973) and interpersonal conflict (IPC) paradigm (Hammond, 1965,
1973; Hammond et al., 1966b). The triple-system lens model for the IPL
and IPC paradigms is shown in Figure 3. In Hammond’s (2001a, p.\ 471)
own view, the triple-system case represented “perhaps the most
important expansion of Brunswik’s lens model. It is possibly one that
Brunswik did not anticipate.”

In both the IPL and IPC paradigms, the judges face a common task, where
there is an environmental criterion with which multiple, proximal cues
are imperfectly associated. Judges attempt to infer the value of the
criterion on the basis of available cues. The IPL and IPC paradigms
share much in common. The main differences lie in the specific elements
of the triple-system case where attention is focused. In the IPL
paradigm, the focus may be either on learning about or learning from
the other person, whereas in the IPC paradigm, the focus is on reducing
disagreement with the other person. We present these two paradigms
below.

Interpersonal learning \textit{about} the other person is focused on
understanding the other person and developing the ability to predict
how he or she will respond to the task. The ability to predict
accurately the other person’s judgments requires an understanding of
the weights, function forms, and organizing principle that the other
person uses to combine information from cues into a judgment.
Prediction accuracy is also constrained by the degree of cognitive
control the other person exercises in making judgments. The other
person’s $R_s$ plays the same role as $R_e$ does in the two-system
case, constraining the upper limit of achievement.

In one of the original interpersonal learning studies, Hammond et al.\
(1966b) found that, on average, paired participants were able to
predict one another’s responses quite well, as well as one another’s
differential cue weights, linear (although not to the extent
hypothesized), and non-linear cue use. Participants were also more
likely to learn about the other person if that person was more
reliable, and through interaction each pair’s policy similarity
increased.

Some research on interpersonal learning about the other person has
focused on the ability to predict another’s judgments when there is no
common environmental criterion, for instance, in negotiation, when the
relevant judgments are about the desirability of potential settlements
(Balke, Hammond \& Meyer, 1973; Miller, 1973; Mumpower, Sheffield,
Darling \& Milter, 2004).

Research on interpersonal learning \textit{from} the other person
focuses on understanding how a judge learns about a task by observing
the judgments that another person makes. Clearly, interpersonal
learning from the other person is not independent of interpersonal
learning about the other person — to learn about the task from another
person, one needs to learn how the other person interacts with the
task. The other person must convey relevant information about the task
for interpersonal learning to be better than individual-only task
learning. 

Hammond (1972) found that function forms affected cue utilization
validities where, for instance, after interpersonal learning,
individuals trained to use a non-linear cue could give up reliance on
that cue and learn to use a linear cue faster than individuals trained
to use a linear cue. In a set of three experiments, Earle (1973)
reported that participants taught to rely solely on linear cues
required interpersonal learning from participants using non-linear cues
in order to switch to using non-linear ones, but not vice versa.
Interpersonal learning was also necessary for learning negative linear
or non-linear rules. 

Research has also investigated the effects of task characteristics on
interpersonal learning and found the effects to be similar to those
found in the multiple-cue probability learning paradigm, with the
exception that non-linear policies are easier to learn through
interpersonal learning (Hammond et al., 1975). Cognitive feedback has
also been shown to be useful in interpersonal learning tasks (Balke et
al., 1973; Miller, 1973). 

Judges may find themselves in false agreement, impeding interpersonal
learning, when they reach similar judgments despite different polices
because of high levels of inter-cue correlations (Mumpower \& Hammond,
1974). On the other hand, lack of cognitive control means that parties
may disagree in practice even though they agree in principle, referred
to as false disagreement (Dhami \& Olsson, 2008). These latter
observations lead us to describe Hammond’s IPC paradigm.

Research in the IPC paradigm is motivated by Hammond’s observation
(1965, 1973; Hammond et al., 1966b) that not all conflict and
disagreement stem from differences in incentives or motivations.
Cognitive conflict may arise due to differences in what people “believe
to be the efficient, just, and moral ways to solve their problems”
(Hammond, 1973, p.\ 189). As Hammond and Grassia (1985, p.\ 233) put it
“People dispute many things besides differential gain, or who gets
what. They also disagree about (a) the facts (what is, what was), (b)
the future (what will be), (c) values (what ought to be), and (d)
action (what to do).” 

As the triple-system lens model in Figure 3 illustrates, there are four
primary potential sources of cognitive conflict. Judges may disagree
about the relative importance (i.e., relative weights) of the proximal
cues; they may disagree about the relationship between values of the
cues and values of the criterion (i.e., the appropriate function forms
for cues); they may disagree about the appropriate rule for combining
information from cues into a judgment (i.e., the organizing principle);
and they may disagree because one or both judges are unable to make
judgments in a consistent way (i.e., imperfect cognitive control).

In a standard interpersonal conflict experiment (Brehmer, 1976; Cooksey,
1996; Hammond, 1965; 1973), participants may be selected either because
they already have conflicting judgment policies or they may be taught
to develop conflicting policies. Unaware that they have different
policies, participants are brought together and asked to co-operate on
solving a set of problems where cues are probabilistically related to
the criterion. On each trial, they consider the available information
and make judgments about the criterion variable individually and then
communicate these to one another. If they disagree they must discuss
the problem until they reach an acceptable joint response. They are
then asked to reconsider their original decisions, and these revisions
remain private. Finally, they are presented with the correct solution.
Conflict or agreement is therefore defined objectively as the actual
differences in the judgments made by the two individuals. 

Conflict may be due to systematic and non-systematic cognitive
differences in the way people perform the task. Systematic differences
refer to features of judgment policies such as the relative cue weights
and non-systematic differences refer to the idea that people may
exercise imperfect cognitive control in the application of their
policies (Mumpower \& Stewart, 1996). Research has shown that, over a
series of trials, participants unlearned their conflicting policies and
developed similar ones, however, conflict persisted because individuals
simultaneously became more inconsistent in applying their revised
policies. Such non-systematic differences accounted for more conflict
than did systematic differences in judgment policies (Brehmer, 1976).
The aforementioned findings have been replicated using different types
of participants and task conditions (Hammond \& Brehmer, 1973). 

Brehmer’s (1976) research demonstrated how the degree and nature of
conflict (i.e., whether it is due to systematic or non-systematic
differences) is affected by task conditions. For example, he showed
that policy consistency is lower in less predictable tasks leading to
less agreement; that non-linear cues lead to lower consistency but do
not affect policy similarity; that when the task contains linear cues
and people only have to use one cue they are more consistent but their
policy similarity is unaffected; that when the task contains both
linear and non-linear cues and people only have to use one cue their
policy similarity and consistency is higher; and that inter-cue
correlations lead to less policy similarity. 

Hammond and Brehmer (1973) applied the technique of cognitive feedback
and developed a cognitive aid to conflict resolution called POLICY
(originally called COGNOGRAPH). This was an interactive computer
program that enabled people to express their policies, compare them,
change them, and discover the effects of such changes on cognitive
conflict. The emphasis is on teaching consistent new policies. It has
been found that cognitive feedback helps to speed conflict reduction
(Balke et al., 1973). 

In sum, research in the IPC paradigm has demonstrated that agreement
could be studied in the same way as achievement. Although the study of
two cognitive systems has become popular, and despite the potential for
theoretical and methodological advance of the concept of cognitive
conflict, Dhami and Olsson’s (2008) review reveals that research on
cognitive conflict using the lens model has declined sharply. It has
been replaced by research on what is called ‘task conflict’ conducted
by scholars interested in group conflict. Unfortunately, the latter has
less theoretical precision and methodological rigor than Hammond’s IPC
paradigm.

\section{Social Judgment Theory}

In 1975, after two decades of work building on Brunswik’s ideas and
advancing them in the study of clinical judgment, multiple-cue
probability learning, interpersonal learning and interpersonal
conflict, Hammond and his colleagues (Hammond et al., 1975) synthesized
the key elements of this work into a unified framework, which they
called social judgment theory (SJT). SJT is not a theory providing
testable hypotheses about the nature of human judgment, but a
meta-theory that provides a framework to guide research in this
endeavor. 

The basic concepts of SJT are all foreshadowed by the two decades of
work that preceded it, and so we shall not discuss them in detail here.
Suffice it to say that in SJT equal status is accorded to both the
human and environment, there are parallel concepts depicting each side
of the lens, and the distinction between surface (proximal cues) and
depth (the environmental criterion and judgment about the environmental
criterion) is essential.

Social judgment theorists, as they have come to be known, study “life
relevant” issues (Hammond et al., 1975, p.\ 276), and so much of SJT
research is applied. There are four basic goals of SJT research: (a) to
analyze judgment tasks and processes; (b) to analyze the structure of
achievement and agreement; (c) to understand how humans learn to
achieve and agree; and (d) to find methods for improving achievement
and agreement (Brehmer \& Joyce, 1988; Hammond et al., 1975). SJT
research, thus, aims to describe behavior before prescribing changes to
improve it. The model of the environment serves as a benchmark,
indicating how judgment can be improved (Brehmer \& Joyce, 1988;
Hammond et al., 1964). Cognitive performance may be enhanced by
cognitive feedback and cognitive (decision) aids (Hammond et al.,
1975). 

Several types of judgment situations are distinguished in SJT research.
These are the double-systems design, single-system design,
triple-systems design, the $N${}-systems design and the
hierarchical design (see also Hammond, 1972; Hammond et al., 1975). The
first refers to Brunswik’s (1952) original lens model as shown in
Figure 1, and involves an analysis of the interaction between an
individual and a task. As mentioned earlier, this framework is used to
study achievement and multiple-cue probability learning. 

The other judgment situations represent modifications to the original
lens model. As mentioned earlier, the single-system design refers
simply to an individual’s judgment policy, with no reference to
achievement in the task (see Figure 2). The triple-systems design
involves one task and two individuals (see Figure 3). It is used to
study interpersonal learning and interpersonal conflict. The
$N${}-systems design involves more than one individual and may
or may not include an analysis of the task. Research on policy
formation is conducted within this framework (e.g., Adelman, Stewart,
\& Hammond, 1975; Stewart \& Gelberd, 1976). Finally, there are
judgment situations in which the cues themselves may be judgments made
at earlier stages of the judgment process either by the same or
different judges. Hammond et al.\ (1975, p.\ 286) refer to such
situations as “hierarchical judgment models”. Here, an outcome
criterion is often unavailable and each stage is analyzed
separately.\footnote{ Stewart (1976) developed a hierarchical
formulation of the lens model equation which made it possible to
isolate the contributions of different sets of variables.} 

In SJT research, judgment data is elicited over a series of trials and
is analyzed at the level of the individual (Hammond et al., 1975).
Hammond et al.\ (1975, p.\ 278) state that “the judgment data are
analyzed in terms of multiple regression statistics.” Thus,
correlational statistics and models such as multiple linear regression
are used to describe and explain cognitive performance. However, as
Brehmer (1979, p.\ 199) pointed out: 
\begin{quote}
A common misunderstanding is that SJT holds that the judgment process
itself operates according to the principles of multiple
regression….just because they use these methods for investigating the
judgment process….Instead, the methods are used to test a series of
hypotheses about the nature of the judgment process, hypotheses about
the nature of cue weights, function forms, combination rules, and
predictability.
\end{quote}

SJT is also committed to representative design defined in terms of
formal situational sampling (Brehmer, 1979; Cooksey, 1996; Hammond et
al., 1975; Hammond \& Wascoe, 1980). Here, all of the relevant cues,
cue values, cue distributions, inter-cue correlations and ecological
validities of cues should be representative of those that exist in the
natural version of the task. Hammond et al.\ (1975) recognized that
under representative conditions, the presence of inter-cue correlations
may make it difficult to ascertain the relative independent effects of
each cue upon judgments. As an alternative to representative design,
they recommended multi-method analyses where techniques such as
predicting each cue from the others and successive omission of cues may
be used.\footnote{ Holaday (1933) had first shown how an exemplary
stimulus could be systematically stripped of its complexity through
‘successive omission’ of cues, and Brunswik (1956) viewed this as a
hybrid design—mixing elements of systematic and representative
design.\par } However, Dhami et al.\ (2004) have revealed that
researchers typically only study single-system cases and do not use any
form of representative design. Consequently, little is known about
judgmental achievement for many tasks, and the lack of representative
stimulus sampling threatens the validity of the findings reported. 

Hammond et al.\ (1975, p.\ 304) stated that “the unique contribution of
SJT has been to bring the theory, quantitative procedures, results of
research, and technological innovations (externalization of judgment
policies by means of interactive computer graphics) to bear on social
policy formation outside the laboratory.” Over the past four decades
since its launch, SJT has inspired hundreds of applied and basic
studies of human judgment (for collections of such studies see Brehmer
\& Joyce, 1988; Dhami et al., 2004; Karelaia \& Hogarth, 2008), as well
as some more recent approaches to the psychology of human judgment and
decision making (e.g., Gigerenzer et al., 1999; Juslin et al., 2000).
One applied area of special note has been medical decision making
(e.g., González-Vallejo, Sorum, Stewart, Chessare \& Mumpower, 1998;
Poses, Cebul, Wigton, et al., 1992; Tape, Kripal \& Wigton, 1992; Way,
Allen, Mumpower et al., 1998; Wigton, 1988; 1996). As Wigton (1996)
observed, SJT is particularly well suited to the study of medical
judgments because they characteristically involve decision making under
uncertainty with inevitable error and an abundance of fallible cues.
SJT has been useful in establishing variation among decision makers’
judgments and in their weighting of clinical information.

\section{Cognitive Continuum Theory}

Later in his career, still faithful to Brunswik’s ideas but going beyond
them, Hammond developed cognitive continuum theory (CCT, Hammond,
1996a, 2000a, 2001a; Hammond et al., 1987). The key propositions of CCT
include the premises that cognition moves on an intuitive-analytical
continuum; that quasi-rationality represents a commonly used and
important middle-ground on this continuum; that cognitive tasks induce
different modes of cognition; and that the upper level of cognitive
performance is dictated by the match between properties of the task and
mode of cognition. 

More specifically, CCT states that there are modes of cognition that lie
in-between intuition and analysis (see also Dhami, Belton \&
Goodman-Delahunty, 2015; Dhami \& Thomson, 2012). Intuition (often also
referred to as System 1, experiential, heuristic, and associative
thinking) is generally considered to be an unconscious, implicit,
automatic, holistic, fast process, with great capacity, requiring
little cognitive effort. By contrast, analysis (often also referred to
as System 2, rational, and rule-based thinking) is generally
characterized as a conscious, explicit, controlled, deliberative, slow
process that has limited capacity and is cognitively demanding. The
modes of cognition that lie in-between are quasirational. 

As Hammond (2010, p.\ 331) points out, the term ‘quasi’ does not mean
that quasirational modes of cognition are the result of “improper
cognitive activity”. In addition, Hammond (1996, pp.\ 166--167, brackets
added) takes pains to differentiate quasirationaity from Herbert
Simon’s (1957) concept of bounded rationality, which he states “means
that cognitive activity has neither the time nor resources to
explore […] completely the “problem space” of the task. The problem space
that \textit{is} explored, however, is explored in a rational or
analytical fashion.” For Hammond, quasirationality is distinct from
rationality. It comprises different combinations of intuition and
analysis, and so may sometimes lie closer to the intuitive end of the
cognitive continuum and at other times closer to the analytic end.

Brunswik (1943, 1952) pointed to the adaptive nature of perception (and
cognition).\footnote{ Hammond (1996) points out that Simon (1991)
similarly viewed humans as ‘adaptive systems’ whose efforts to
understand their environments are constrained, and consequently limit
any potential for optimization. } For Hammond (1996a, 2000b), modes of
cognition are determined by properties of the task (and/or expertise
with the task). Task properties include, for example, the amount of
information, its degree of redundancy, format, and order of
presentation, as well as the decision maker’s familiarity with the
task, opportunity for feedback, and extent of time pressure. The
cognitive mode induced will depend on the number, nature and degree of
task properties present. A task comprising either intermediate levels
of, or a combination of, those properties inducing pure intuition or
pure analysis will induce quasirationality. Depending on task
properties, quasirationality may imply a combination where there is
greater use of intuition than analysis, or vice versa. Movement along
the cognitive continuum is characterized as oscillatory or alternating,
thus allowing different forms of compromise between intuition and
analysis (i.e., different forms of quasirationality). 

According to Hammond, cognitive tasks can be differentiated from one
another with regard to their properties as well as the mode of
cognition they induce (see also Dhami \& Thomson, 2012). Studies
support the idea that different task properties induce different modes
of cognition (Dunwoody, Haarbauer, Mahan, Marino \& Chu-Chun, 2000;
Hamm, 1988; Hammond et al., 1987; see also Mahan, 1994). Success on a
task inhibits movement along the cognitive continuum (or change in
cognitive mode) while failure stimulates it. Hamm (1988) additionally
demonstrated that cognitive mode can shift during a task. 

Brunswik (1943, 1952, 1956) stressed the significance of the
correspondence or fit between properties of the task and the mode of
cognition. Hammond (1988) predicted that judgment performance is
contingent on the degree of correspondence between task properties and
mode of cognition. The key implication is that pure analysis may be
neither necessary nor sufficient for ceiling-level performance.
Evidence suggests that task characteristics are important in
determining the upper bound of performance (e.g., Seifert \& Hadida,
2013; Rusou, Zakay \& Usher, 2013) and that achievement is greater when the cognitive mode matches
that induced by the task (e.g., Dunwoody et al., 2000; Hammond et al.,
1987).

Although there is a growing body of evidence on the nature and
performance of intuitive versus analytic cognition (e.g., Dunwoody et
al., 2000; Hammond et al., 1987; Mahan, 1994; Marewski \& Mehlhorn,
2011), there is a distinct dearth of research on the operation and
outcomes of quasirationality, even though, as some have argued, this is
the most common mode of cognition in many consequential tasks (Dhami et
al., 2015; Dhami \& Thomson, 2012). In their efforts to identify the
processes involved in intuitive versus analytic cognition, Glöckner and
his colleagues have found some similarities and differences between
these two modes of cognition (e.g., Glöckner \& Betsch, 2008a; 2008b;
2012; Jekel, Glöckner, Fiedler \& Bröder, 2012; Horstmann, Ahlgrimm,
\& Glöckner, 2009). Their findings suggest that intuition and analysis
may operate in an integrative fashion and thus potentially shed light
on different forms of quasirationality. For instance, quasirationality
may allow individuals to use a lot of information quickly. Other work
measuring the performance of different modes of cognition, for example
by Blattberg and Hoch (1990), has demonstrated that a quasirational
model which combined managerial intuition (expertise) and statistical
analysis repeatedly outperformed purely intuitive and statistical
models in five forecasting tasks (see also Ganzach, Kluger \& Klayman,
2000).

Hammond (2007) returned to the themes of analysis and intuition and the
cognitive continuum in his last book entitled \textit{Beyond
Rationality: The Search for Wisdom in a Troubled Time}, published at
age 92. At the heart of his argument is the proposition that the key to
wisdom lies in being able to match modes of cognition to properties of
the task. In brief, for most judgment and decision making tasks,
quasirationality is what is required for wisdom. According to Hammond
(2007, p.\ 237), “…the tactics that most of us use most of the time are
neither fully intuitive nor fully analytical: they are a compromise
that contains some of each; how much of each depends on the nature of
the task and on the knowledge the person making the judgment brings to
the task.”

\section{Solving Applied Problems}

Finally, it is notable that Hammond was as comfortable conducting
basic research in the laboratory as he was in solving applied
problems, especially in the policy context. In fact, an emphasis on
application to social policy formation was part of the original
declaration of SJT (Hammond et al., 1975). A number of SJT inspired
policy applications were undertaken on topics ranging from citizen
participation in community planning, through water resource planning,
to air pollution management, and so forth (e.g., Brady \& Rappoport,
1973; Flack \& Summers, 1971; Hammond, Mumpower \& Smith, 1977;
Mosier, Skitka, Heers \& Burdick, 1998; Mumpower, Veirs \& Hammond,
1979; Stewart \& Gelberd, 1972). Perhaps the best known and most
influential of these applications is Hammond and Adelman’s (1976)
‘Denver Bullet Study.’

The policy issue in the Denver Bullet Study concerned the type of bullet
that should be used by the Denver (Colorado, USA) City Police. In 1974,
the Denver Police Department (DPD) decided to change its handgun
ammunition because it was argued that conventional round-nosed bullets
provided insufficient ‘stopping effectiveness’ (i.e., the ability to
incapacitate and thus to prevent the person shot from firing back at
the police or others). The DPD chief recommended using a hollow-point
bullet, claiming that such bullets flattened on impact, thus decreasing
penetration, increasing stopping effectiveness, and decreasing ricochet
potential. This claim was challenged by, for example, the American
Civil Liberties Union and minority groups. Opponents stated that the
new bullets were in fact outlawed ‘dum-dum’ bullets which were more
injurious than the round-nosed bullet and so should be barred from use.
There were public hearings, debates and disputes, and appeals by both
sides to ballistics experts for scientific information and support.
Disputants focused on evaluating the merits of specific alternative
bullets—confounding the physical effect of the bullets with social
policy implications. As Hammond and Adelman (1976) realized, they
confounded questions of value — what the bullet should accomplish with
questions of fact — concerning ballistic characteristics of specific
bullets. Arguments favored one option or another, but obscured the
basis for a preference.

Hammond and Adelman (1976) stated that policy makers inadvertently had
adopted the role of (unqualified) ballistics experts, and ballistics
experts inadvertently had adopted the role of (poor) policy makers.
Hammond and Adelman intervened to first discover the important policy
dimensions from the policy makers’ viewpoint and then elicited
ballistics experts’ ratings of the bullets on these dimensions. The
relevant dimensions were stopping effectiveness, probability of serious
injury, and probability of harm to bystanders. The experts’ ratings of
the bullets on the last two dimensions were almost perfectly
confounded. The probability of serious harm to bystanders is highly
related to the penetration of the bullet, whereas the probability of
the bullet effectively stopping someone from returning fire is highly
related to the width of the entry wound. Giving equal weights to these
dimensions, and combining these weights with the experts’ technical
judgments, led Hammond and Adelman (1976, p.\ 395) to identify a bullet
that “has greater stopping effectiveness and is less apt to cause
injury (and is less apt to threaten bystanders) than the standard
bullet then in use by the DPD.” The bullet they recommended was
accepted by both the DPD and the Denver City Council and put into
operation. 

According to Adelman (1988, p.\ 443), the above study illustrates the
significant contribution of SJT in identifying the importance of the
“separation of facts and values” in resolving social policy disputes.
Adelman identifies five key points to be derived from the Denver Bullet
Study and similar applications. First, social polices comprise three
types of judgments: (a) value judgments about what ought to be, (b)
factual judgments about what is or will be, and (c) evaluative
judgments that integrate value and factual judgments into a final
policy decision. Second, policy makers should be responsible for value
judgments and technical experts for factual judgments. Third, methods
exist to build quantitative models for both value and factual
judgments. Fourth, analytical methods can and should be used to combine
value and factual judgments so that alternatives can be systematically
evaluated. Fifth, cognitive feedback can be used to make the
implications of value judgments and factual judgments explicit.

In 1996, Hammond published a book entitled \textit{Human Judgment and
Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable
Injustice} which attempted to understand the policy formation
process.\footnote{ This book won the 1997 Outstanding Research
Publication Award from the American Educational Research Association.}
He included Brunswikian and SJT research as well as other work,
notably, Ward Edwards’ (1954, 1961) contributions using Bayesian-themed
behavioral decision theory. This 1996 book emphasized two key themes
that Hammond had been considering for a number of years, but which he
was able to more fully address here. The first theme was the
distinction between theories of truth that emphasized
\textit{coherence} competence and theories of truth that emphasized
\textit{correspondence} competence. The former is exemplified in Ward
Edward’s Bayesian approach (and later in Kahneman and Tversky’s
‘heuristics and biases’ approach, e.g., Kahneman, Slovic \& Tversky,
1982) which accentuate normative rationality defined in terms of
internal and logical consistency. The latter is exemplified by the
Brunswikian and SJT approaches in which empirical accuracy defines
achievement. The issue, according to Hammond, was whether in a policy
context, it was more important to be rational (internally and logically
consistent) or to be empirically accurate. Hammond’s treatment of the
strengths and limitations of the two approaches is too nuanced and
sophisticated to review fully here, but his conclusions are guardedly
optimistic that in the realm of policy we can strike a balance between
coherence and correspondence. The key to achieving this balance lies in
how we think about error, which was the second theme.

Hammond (1996a) emphasized the duality of error. He noted that, although
this concept had been recognized long before, it was not until Neyman
and Pearson (1933) that duality of error (Type I and Type II error) was
formally introduced with a mathematical treatment. The concern with
error was also an important part of the Brunswikian legacy. Brunswik
(1956) demonstrated that the error distributions for intuitive and
analytical processes were quite different. Intuitive processes led to
distributions in which there were few precisely correct responses but
also few large errors, whereas with analysis there were often many
precisely correct responses but occasional large errors. According to
Hammond, duality of error inevitably occurs whenever decisions must be
made in the face of irreducible uncertainty, or uncertainty that cannot
be reduced at the moment action is required. Thus, there are two
potential mistakes that may arise — false positives (Type I errors) and
false negatives (Type II errors) — whenever policy decisions involve
dichotomous choices, such as whether to admit or reject college
applications, claims for welfare benefits, and so on. Hammond (1996a)
argued that any policy problem involving irreducible uncertainty has
the potential for dual error, and consequently unavoidable injustice in
which mistakes are made that favor one group over another. 

% reader?

In this work, Hammond made a remarkable and influential advance to the
Brunswikian and SJT approaches, pushing those ideas all the way from
perception through thinking to policy formation. He identified two
tools of particular value for analyzing policy making in the face of
irreducible environmental uncertainty and duality of error. These were
Signal Detection Theory (SDT; Tanner \& Swets, 1954; Swets, 1992) and
the Taylor-Russell (1939) paradigm. Others have extended these ideas in
a number of substantive policy areas such as college admissions
(Mumpower, Nath \& Stewart, 2002), child welfare services (Mumpower,
2010; Mumpower \& McClelland, 2014), international policy (Dunwoody \&
Hammond, 2006), and mammography (Stewart \& Mumpower, 2004). In fact,
Hammond’s 1996 book, published almost 50 years after his first academic
paper, is now his most cited work. 

\renewcommand{\baselinestretch}{1.025} \large\normalsize

\section{Closing Remarks}

In closing, it is worth noting that Hammond made several attempts,
albeit with mixed success, to integrate Brunswikian and SJT
perspectives with other normative, prescriptive, and descriptive
theories of human judgment and decision making. One of the first
efforts in this regard was the book published in 1980 entitled
\textit{Human Judgment and Decision Making: Theories, Methods, and
  Procedures} (Hammond, McClelland \& Mumpower, 1980). This grew out
of the Twelfth Annual Conference on Human Judgment, held in April
1978, in Boulder, Colorado. The conference Hammond organized brought
together 25 prominent scholars who represented approaches to the study
of judgment and decision making that focused more on analytic thought
to those that focused more on intuition.\footnote{ For a photograph of
  attendees at the conference, who included pioneers of judgment and
  decision making research such as Norman Anderson, Hutton Baron,
  Robyn Dawes, Michael Doherty, Ward Edwards, Hillel Einhorn, Baruch
  Fischhoff, Kenneth Hammond, Marty Kaplan, Ralph Keeney, Sarah
  Lichtenstein, Ned Jones, Mel Manis, Gary McClelland, Lee Ross, James
  Shanteau, Kelly Shaver, Paul Slovic, Thomas Stewart, Amos Tversky,
  and others, see
  \url{http://www.brunswik.org/photos/bldrgroup.html}.} Six approaches
were included as follows: Decision Theory, Behavioral Decision Theory,
Psychological Decision Theory, Social Judgment Theory, Information
Integration Theory, and Attribution Theory. These were compared and
contrasted in terms of six categories: origins, scope, intended
function, principal concepts, loci of concepts, and intended uses. The
book remains widely cited, although it is unclear how much direct
influence it had on the field of judgment and decision making, and
some reviewers were unconvinced that integration was possible (e.g.,
Phillips, 1982).

Hammond was not deterred from continuing his efforts to explicate his
own work and the work of others in the field of judgment and decision
making. In 1981, he was part of a team that produced a glossary
intended to help decipher the idiosyncratic terminology used by
different approaches and disambiguate usage of the same or similar
terms to mean different things (Anderson, Deane, Hammond et al., 1981).
The team defined each major concept in judgment and decision making,
and provided its source, usage, current status and relationship with
other~concepts. Indeed, students entering the field and scholars
outside the field still find this glossary useful, although it now
needs to be updated to encompass the continued evolution and addition
of approaches over the decades since the book was published.

In this paper, we have discussed the major and distinctive contributions
that Hammond made to the science of human judgment and decision making
throughout his illustrious career. In sum, his ground-breaking research
extended Egon Brunswik’s theory of probabilistic functionalism and lens
model framework to the study of human judgment and decision making.
Through SJT, Hammond proposed a general framework for the study of
judgment processes. This enabled analysis of individual judgment as
well as judgment in situations where individuals interact with one
another. This framework also allowed researchers to describe and
explain judgmental achievement as well as explore ways to improve it.
Going beyond Brunswik’s ideas, Hammond introduced CCT which is the
first comprehensive theory in psychology of the relation between task
properties and cognition. Hammond’s commitment to scholarship was
matched by his desire to improve our social world; evidenced by his
efforts to apply research to the improvement of public policy
decisions.

The importance of Hammond’s contribution to the psychology of judgment
and decision making continues to be manifest in the work of his
students and colleagues in such disparate fields as educational
research, clinical decision making, healthcare and medical decision
making, social work, social perception, group decision making and
negotiation, accounting and auditing, law, public policy analysis,
human factors, and human technology interaction. Like Hammond, many of
these researchers have engaged in both basic and applied research. We
have to thank Hammond for creating a path on which we could travel
during our academic careers. We hope the present paper will inspire new
scholars to take their own intellectual journeys building upon, and
advancing, Hammond’s legacy.

\renewcommand{\baselinestretch}{1.00} \large\normalsize

\section*{Brief Biography of Kenneth R. Hammond}

Kenneth R. Hammond was born in San Francisco, California in 1917. He
died in Boulder, Colorado, in 2015. He received his BA, MA, and PhD
from the University of California, Berkeley. In 1948, he accepted an
appointment in the Department of Psychology at the University of
Colorado, Boulder, where he taught until his retirement in 1987. There,
he co-founded both the Institute of Behavioral Science and the Center
for Research on Judgment and Policy. He was a visiting professor at the
Universities of Hawaii, Berkeley, and Arizona, as well as a visiting
scholar at the International Institute for Applied Systems Analysis and
the Rockefeller Foundation Bellagio Center. His research was supported
by the National Science Foundation, the U. S. Public Health Service,
the Army Research Institute, the Office of Naval Research, the
Commonwealth Fund, and other private foundations. In 1982, he was
awarded an honorary doctorate from the University of Uppsala, Sweden.
In 1983, he founded the Brunswik Society which continues to serve as a
forum for the discussion of Brunswikian ideas among researchers from
around the world. In 1987--88, Hammond served as the second President of
the Society for Judgment and Decision Making, a Society that some
recall he helped to establish.

\section*{References}
\begin{hangparas}{1em}{1}

  Adelman, L. (1988). Separation of facts and values. In B. Brehmer \&
  C.  R. B. Joyce (Eds.), \textit{Human judgment: The SJT view} (pp.\
  443--464). New York: North-Holland.

Adelman, L., \& Mumpower, J. L. (1979). The analysis of expert judgment.
\textit{Technological Forecasting and Social Change}, \textit{15},
191--204.

Adelman, L., Stewart, T. R, \& Hammond, K. R. (1975). A case history of
the application of social judgment theory in policy formation.
\textit{Policy Sciences}, \textit{6}, 137--159.

Anderson, B. F., Deane. D. H., Hammond, K. R., McClelland. G. H., \&
Shanteau. J. C. (1981). \textit{Concepts in judgment and decision
research: Definitions, sources, interrelations, comments}. New York:
Praeger. 

Balke, W. M., Hammond, K. R., \& Meyer, G. D. (1973). An alternate
approach to labor-management relations. \textit{Administrative Science
Quarterly}, \textit{18}, 311--327. 

Balzer, W. K., Doherty, M. E., \& O'Connor, R. (1989).
Effects of cognitive feedback on performance. \textit{Psychological
Bulletin}, \textit{106}(3), 410--433. 

Blattberg, R. C., \& Hoch, S. J. (1990). Database models and managerial
intuition: 50\% model+ 50\% manager. \textit{Management Science},
\textit{36}(8), 887--899.

Bottenberg, R. A., \& Christal, R. E. (1961). \textit{An iterative
technique for clustering criteria which retains optimum predictive
efficiency}. (WADD-TN-61--30, ASTIA Document AD-261 615). Lackland Air
Force Base, TX: Personnel Research Laboratory. Wright Air Development
Division.

Brady, D., \& Rappoport, L. (1973). Policy capturing in the field: The
nuclear safeguards problem. \textit{Organizational Behavior and Human
Performance}, \textit{9}(2), 253--266.

Brehmer, B. (1970). Inference behavior in a situation where the cues are
not reliably perceived. \textit{Organizational Behavior and Human
Performance}, \textit{5}, 330--347.

Brehmer, B. (1972). Cue utilization and cue configurations in
multiple-cue probability learning. \textit{Organizational Behavior and
Human Performance}, \textit{8}, 286--296.

Brehmer, B. (1973a). Effects of task predictability and cue validity on
interpersonal learning of inference tasks involving both linear and
nonlinear relations. \textit{Organizational Behavior and Human
Performance}, \textit{10}, 24--46. 

Brehmer, B. (1973b). Effects of cue validity on interpersonal learning
of inference tasks with linear and nonlinear cues. \textit{American
Journal of Psychology}, \textit{86}, 29--48.

Brehmer, B. (1974). Effects of cue validity and task predictability on
interpersonal learning of linear inference tasks.
\textit{Organizational Behavior and Human Performance}, \textit{12}(1),
17--29.

Brehmer, B. (1976). Social judgment theory and the analysis of
interpersonal conflict. \textit{Psychological Bulletin},
\textit{83}(6), 985--1003.

Brehmer, B. (1979). Preliminaries to a psychology of inference.
\textit{Scandinavian Journal of Psychology}, \textit{20}, 193--210.

Brehmer, B. (1980). In one word: Not from experience. \textit{Acta
Psychologica}, \textit{45}, 223--241. 

Brehmer, B. (1994). The psychology of linear judgment models.
\textit{Acta Psychologica}, \textit{87}, 137--154.

Brehmer, A., \& Brehmer, B. (1988). What have we learned about human
judgment from thirty years of policy capturing. In B. Brehmer, \& C. R.
B. Joyce (Eds.), \textit{Human judgment: The SJT view} (pp.\ 75--114).
New York: North-Holland.

Brehmer, B., \& Joyce, C. R. B. (1988). \textit{Human judgment: The SJT
view}. New York: North-Holland.

Brunswik, E. (1940). Thing constancy as measured by correlation
coefficients. \textit{Psychological Review}, \textit{47}, 69--78.

Brunswik, E. (1943). Organismic achievement and environmental
probability. \textit{Psychological Review}, \textit{50}, 255--272.

Brunswik, E. (1944). Distal focussing of perception: Size constancy in a
representative sample of situations. \textit{Psychological Monographs,
56}(254), 1--49. 

Brunswik, E. (1952). \textit{The conceptual framework of psychology}.
Chicago, IL: University of Chicago Press.

Brunswik, E. (1955a). In defense of probabilistic functionalism: A
reply. \textit{Psychological Review}, \textit{55}, 236--242. 

Brunswik, E. (1955b). “Ratiomorphic” models of perception and thinking.
\textit{Acta Psychologica}, \textit{11}, 108--109. 

Brunswik, E. (1955c). Representative design and probabilistic theory in
a functional psychology. \textit{Psychological Review}, \textit{55},
193--217. 

Brunswik, E. (1956). \textit{Perception and the representative design of
experiments}. Berkeley, CA: University of California Press.

Brunswik, E. (1957). Scope and aspects of the cognitive problem. In H.
Gruber, K. R. Hammond, \& R. Jessor (Eds.), \textit{Contemporary
approaches to cognition} (pp.\ 5--31). Cambridge, MA: Harvard University
Press.

Camerer, C. F. (1981).  General conditions for the success of
bootstrapping models.  \textit{Organizational Behavior and Human
Performance, 27}, 411--422.

Castellan, N. J. (1973). Comments on the “lens model” equation and the
analysis of multiple-cue judgment tasks. \textit{Psychometrika},
\textit{38}, 87--100.

Castellen, N. J. (1992). Relations between linear models: Implications
for the lens model. \textit{Organizational Behavior and Human Decision
Processes, 51, }364--381.

Christal, R. E. (1963). \textit{JAN: A technique for analyzing group
judgment}. (TDR-63--3 AD-402 813). Lackland Air Force Base, TX:
Personnel Research Laboratory. Aerospace Medical Division.

Cook, R. L., \& Stewart, T. R. (1975). A comparison of seven methods
for obtaining subjective descriptions of judgmental
policy. \textit{Organizational Behavior and Human Performance, 123},
31--45.

Cooksey, R. W. (1996). \textit{Judgment analysis: Theories, methods, and
applications}. New York: Academic Press.

Cooksey, R. W., \& Freebody, P. (1985). Generalized multivariate lens
model analysis for complex human inference tasks.
\textit{Organizational Behavior and Human Performance}, \textit{35},
46--72.

Deane, D. H., Hammond, K. R., \& Summers, D. (1972). Acquisition and
application of knowledge in complex inference tasks. \textit{Journal of
Experimental Psychology}, \textit{92}, 20--26.

Dhami, M. K. (2011). Representative design: A challenge for scientific
psychology. In K. O. Moore, \& N. P. Gonzalez (Eds.), \textit{Handbook
on psychology of decision-making}. Nova Science Publishers, Inc.

Dhami, M. K., \& Ayton, P. (2001). Bailing and jailing the fast and
frugal way. \textit{Journal of Behavioral Decision Making},
\textit{14}(2), 141--168.

Dhami, M. K., Belton, I., \& Goodman-Delahunty, J. (2015). Quasirational
models of sentencing. \textit{Journal of Applied Research in Memory and
Cognition}, \textit{4}(3), 239--247.

Dhami, M. K., \& Harries, C. (2001). Fast and frugal versus regression
models of human judgement. \textit{Thinking \& Reasoning},
\textit{7}(1), 5--27.

Dhami, M. K., \& Harries, C. (2010). Information search in heuristic
decision making. \textit{Applied Cognitive Psychology, 24}(4), 571--586.

Dhami, M. K., Hertwig, R., \& Hoffrage, U. (2004). The role of
representative design in an ecological approach to cognition.
\textit{Psychological Bulletin}, \textit{130}(6), 959--988.

Dhami, M. K., \& Olsson, H. (2008). Evolution of the interpersonal
conflict paradigm. \textit{Judgment and Decision Making, 3}(7),
547--569. 

Dhami, M. K., \& Thomson, M. E. (2012). On the relevance of Cognitive
Continuum Theory and quasirationality for understanding management
judgment and decision making. \textit{European Management Journal,
30}(4), 316--326.

Doherty, M. E., \& Balzer, W. K. (1988). Cognitive feedback. In B.
Brehmer \& C. R. B. Joyce (Eds.), \textit{Human judgment: The SJT view
}(pp.\ 163--197). Amsterdam, Netherlands: North Holland. 

Dunwoody, P. T., Haarbauer, E., Mahan, R. P., Marino, C., \& Chu-Chun,
T. (2000). Cognitive adaptation and its consequences: A test of
cognitive continuum theory. \textit{Journal of Behavioral Decision
Making}, \textit{13}(1), 35--54. 

Dunwoody, P. T., \& Hammond, K. R. (2006). The policy of pre-emption and
its consequences: Iraq and beyond. \textit{Peace and Conflict: Journal
of Peace Psychology}, \textit{12}(1), 19--35. 

Earle, T. C. (1973). Interpersonal learning. In L. Rappoport \& D.
Summers (Eds.), \textit{Human judgment and social interaction }(pp.
240--266). New York: Holt, Rinehart, \& Winston.

Edwards, W. (1954). The theory of decision making. \textit{Psychological
Bulletin}, \textit{41}, 380--417. 

Edwards, W. (1961). Behavioral decision theory. \textit{Annual Review of
Psychology}, \textit{12}(1), 473--498.

Flack, J. E., \& Summers, D. A. (1971). Computer aided conflict
resolution in water resource planning: An illustration. \textit{Water
Resources Research}, 7, 1410--1414. 

Ganzach, Y., Kluger, A. N., \& Klayman, N. (2000). Making decisions from
an interview: Expert measurement and mechanical combination.
\textit{Personnel Psychology}, \textit{53}(1), 1--20. 

Gigerenzer, G. (2001). Ideas in exile: The struggles of an upright man.
In K. R. Hammond \& T. R. Stewart (Eds.), \textit{The essential
Brunswik} (pp.\ 445--452). Oxford: Oxford University Press.

Gigerenzer, G., \& Goldstein, D. G. (1996). Reasoning the fast and
frugal way: models of bounded rationality. \textit{Psychological
Review}, \textit{103}(4), 650--659.

Gigerenzer, G., Hertwig, R., \& Pachur, T. (Eds.),
(2011). \textit{Heuristics: The foundations of adaptive
behavior}. New York: Oxford University Press.

Gigerenzer, G., Hoffrage, U., \& Kleinbolting, H. (1991). Probabilistic
mental models: A Brunswikian theory of confidence.
\textit{Psychological} \textit{Review, 98, }506--528.

Gigerenzer, G., Todd, P. M., \& ABC Research Group, T. (1999).
\textit{Simple heuristics that make us smart}. New York: Oxford
University Press.

Gillis, J. S., Lipkin, J. O., \& Moran, T. J. (1981). Drug therapy
decisions: A social judgment analysis. \textit{The Journal of Nervous
and Mental Disease}, \textit{169}(7), 439--447.

Glöckner, A., \& Betsch, T. (2008a). Modeling option and strategy
choices with connectionist networks: Towards an integrative model of
automatic and deliberate decision making. \textit{Judgment and Decision Making,
3}, 215--228. 

Glöckner, A., \& Betsch, T. (2008b). Multiple-reason decision making
based on automatic processing. \textit{Journal of Experimental Psychology:
Learning, Memory, and Cognition, 34}, 1055--1075. 

Glöckner, A., \& Betsch, T. (2012). Decisions beyond boundaries: When
more information is processed faster than less. \textit{Acta
Psychologica, 139}, 532--542.

González-Vallejo, C., Sorum, P. C., Stewart, T. R., Chessare, J. B., \&
Mumpower, J. L. (1998). Physicians' diagnostic
judgments and treatment decisions for acute otitis media in children.
\textit{Medical Decision Making}, \textit{18}(2), 149--162.

Grebstein, L. (1963). Relative accuracy of actuarial prediction,
experienced clinicians, and graduate students in a clinical judgment
task. \textit{The Journal of Consulting Psychology, 37}, 127--132.

Hamm, R. M. (1988). Clinical intuition and clinical analysis: Expertise
and the cognitive continuum. In J. Dowie \& A. Elstein (Eds.),
\textit{Professional judgment: A reader in clinical decision making
}(pp.\ 78--105). Cambridge: Cambridge University Press.

Hammond, K. R. (1948). Subject and object sampling.
\textit{Psychological Bulletin}, \textit{45}, 530--533.

Hammond, K. R. (1954). Representative design vs.\ systematic design in
clinical psychology. \textit{Psychological Bulletin}, \textit{51},
150--159.

Hammond, K. R. (1955). Probabilistic functioning and the clinical
method. \textit{Psychological Review, 62}, 255--262.

Hammond, K. R. (1965). New directions in research on conflict
resolution. \textit{Journal of Social Issues, 21}, 44--66.

Hammond, K. R. (1966). Probabilistic functionalism: Egon Brunswik’s
integration of history, theory, and method of psychology. In K. R.
Hammond (Ed.), \textit{The psychology of Egon Brunswik }(pp.\ 15--80).
New York: Holt, Rinehart, \& Winston.

Hammond, K. R. (1971). Computer graphics as an aid to learning.
\textit{Science, 172}, 903--908.

Hammond, K. R. (1972). Inductive knowing. In J. Royce \& W. Rozeboom
(Eds.), \textit{The psychology of knowing} (pp.\ 285--320). New York:
Gordon and Breach.

Hammond, K. R. (1973). The cognitive conflict paradigm. In L. Rappoport
\& D. Summers (Eds.), \textit{Human judgment and social interaction
}(pp.\ 188--205). New York: Holt, Rinehart, \& Winston.

Hammond, K. R. (1988). \textit{Judgment and decision making in dynamic
tasks}. University of Colorado, Boulder, Center for Research on
Judgment and Policy. Retrieved July 31, 2017, from
\url{http://www.dtic.mil/get-tr-doc/pdf?AD=ADA199907}.

Hammond, K. R. (1996a). \textit{Human judgment and social policy:
Irreducible uncertainty, inevitable error, unavoidable injustice}. New
York: Oxford University Press. 

Hammond, K. R. (1996b). Upon reflection. \textit{Thinking and
Reasoning}, 2, 239--248.

Hammond, K. R. (2000a). Coherence and correspondence theories in
judgment and decision making. In T. Connolly, H. R. Arkes, \& K. R.
Hammond (Eds.), \textit{Judgment and decision making: An
interdisciplinary reader}, 53--65. 

Hammond, K. R. (2000b). \textit{Judgments under stress}. New York:
Oxford University Press.

Hammond, K. R. (2001a). Expansion of Egon Brunswik’s psychology,
1955--1995. In K. R. Hammond \& T. R. Stewart (Eds.), \textit{The
essential Brunswik} (pp.\ 464--480). Oxford: Oxford University Press. 

Hammond, K. R. (2001b). Explaining to the nobility. In K. R. Hammond \&
T. R. Stewart (Eds.), \textit{The essential Brunswik} (pp.\ 134--135).
Oxford: Oxford University Press. 

Hammond, K. R. (2001c). Representative design in action in the middle of
the twentieth century. In K. R. Hammond \& T. R. Stewart (Eds.),
\textit{The essential Brunswik} (pp.\ 67--68). Oxford: Oxford University
Press. 

Hammond, K. R. (2007). \textit{Beyond rationality: The search for wisdom
in a troubled time}. Oxford: Oxford University Press.

Hammond, K. R. (2010). Intuition, no! …quasirationality, yes!
\textit{Psychological inquiry, 21}(4), 327--337.

Hammond, K. R., \& Adelman, L. (1976). Science, values, and human
judgment. \textit{Science, 194}, 389--396.

Hammond, K. R., \& Boyle, P. J. R. (1971). Quasirationality, quarrels,
and new conceptions of feedback. \textit{Bulletin of the British
Psychological Society}, \textit{24}, 103--113.

Hammond, K. R., \& Brehmer, B. (1973). Quasi-rationality and distrust:
Implications for international conflict. In L. Rappoport \& D. Summers
(Eds.), \textit{Human judgment and social interaction }(pp.\ 338--391).
New York: Holt, Rinehart \& Winston.

Hammond, K. R. \& Grassia, J. (1985). The cognitive side of conflict:
From theory to resolution of policy disputes. In S. Oskamp (Ed.),
\textit{Applied social psychology annual. Vol. 6 International conflict
and national public policy issues }(pp.\ 233--254). Beverly Hills, CA:
Sage. 

Hammond, K. R., Hamm, R. M., Grassia, J., \& Pearson, T. (1987). Direct
comparison of the efficacy of intuitive and analytical cognition in
expert judgment. \textit{IEEE Transactions on Systems, Man, and
Cybernetics, 17}, 753--770.

Hammond, K. R., Hursch, C. J., \& Todd, F. J. (1964). Analyzing the
components of clinical inference. \textit{Psychological Review, 71},
438--456.

Hammond, K. R., \& Lang, Z. R. (2017). Confronting inductive inference
(at last): Concepts from aeronautical engineering can lead to advances
in social psychology. Paper presented at 25th International Meeting of
the Brunswik Society, Vancouver, British Columbia, Canada, November.

Hammond, K. R., McClelland, G. H., \& Mumpower, J. (1980). \textit{Human
judgment and decision making: Theories, methods, procedures}. New York:
Praeger.

Hammond, K. R., Mumpower, J. L., \& Smith, T. R. (1977). Linking
environmental models with models of human judgment: A symmetrical
decision aid. \textit{IEEE Transactions on Systems, Man, and
Cybernetics}, \textit{7}, 358--367.

Hammond, K. R., \& Stewart, T. R. (2001a). Introduction. In K. R.
Hammond \& T. R. Stewart (Eds.), The essential Brunswik: Beginnings,
Explications, Applications (pp.\ 3--11). New York: Oxford University
Press.

Hammond, K. R., \& Stewart, T. R. (2001b). \textit{The essential
Brunswik}. Oxford: Oxford University Press. 

Hammond, K. R., Stewart, T. R., Brehmer, B., \& Steinmann, D. O. (1975).
Social judgment theory. In M. Kaplan \& S. Schwartz (Eds.),
\textit{Human judgment and decision processes} (pp.\ 271--312). San
Diego, CA: Academic Press.

Hammond, K. R., \& Summers, D. A. (1965). Cognitive dependence on linear
and non-linear cues. \textit{Psychological Review}, \textit{72},
215--224.

Hammond, K. R., \& Summers, D. A. (1972). Cognitive control.
\textit{Psychological Review, 79}, 58--67.

Hammond, K. R., Summers, D. A., \& Deane, D. H. (1973). Negative effects
of outcome feedback in multiple cue probability learning.
\textit{Organizational Behavior and Human Performance, 9}, 30--34.

Hammond, K. R., \& Wascoe, N. (1980). \textit{Realizations of Brunswik’s
representative design. }San Francisco: Jossey-Bass\textit{.}

Hammond, K. R., Wilkins, M. M., \& Todd, F. J. (1966a). A research
paradigm for the study of interpersonal learning. \textit{Psychological
Bulletin, 65}, 221--232.

Hammond, K. R., Todd, F. J., Wilkins, M. M., \& Mitchell, M. (1966b).
Cognitive conflict between persons: Applications of the “lens model”
paradigm. \textit{Journal of Experimental Social Psychology, 2},
343--360. 

Harvey, N. (2011). Learning judgment and decision making from feedback.
In M. K. Dhami, A. Schlottmann, \& M. Waldmann (Eds.). \textit{Judgment
and decision making as a skill: Learning, development, and evolution}
(pp.\ 406--464). Cambridge: Cambridge University Press.

Herring, F. H. (1954). A psychological study of patient response during
surgery and anesthesia. Unpublished doctoral dissertation, University
of Colorado.

Holaday, B. E. (1933). Die Gro¨\_enkonstanz der Sehdinge bei Variation
der inneren und a¨u\_eren Wahrnehmungsbedingungen [Size constancy for
visual objects under variation of internal and external perceptual
conditions]. \textit{Achiv fu¨r die Gesamte Psychologie, 88, }419--486. 

Holzworth, R. J. (1999). Annotated Bibliography of Cue Probability
Learning Studies. The Brunswik Society. Retrieved on July 6, 2017, from
\url{http://www.brunswik.org/resources/mcplbib.doc}.

Holzworth, R. J., \& Doherty, M. E. (1976). Feedback effects in a metric
multiple cue probability learning task. \textit{Bulletin of the
Psychonomic Society, 8}, 1--3.

Horstmann, N., Ahlgrimm, A., \& Glöckner, A. (2009). How distinct are
intuition and deliberation? An eye-tracking analysis of
instruction-induced decision modes. \textit{Judgment and Decision
Making, 4,} 335--354.

Hursch, C. J., Hammond, K. R., \& Hursch, J. (1964). Some methodological
considerations in multiple-cue probability studies.
\textit{Psychological Review, 71}, 42--60.

Jekel, M., Glöckner, A., Fiedler, S., \& Bröder, A. (2012). The
rationality of different kinds of intuitive decision
processes. \textit{Synthese, 189}, 147--160.

Juslin, P., Winman, A., \& Olsson, H. (2000). Naive empiricism and
dogmatism in confidence research: A critical examination of the
hard-easy effect. \textit{Psychological Review, 107, }384--396.

Kirlik, A. (Ed.), \textit{Adaptive perspectives on human-technology interaction}. Oxford: 
Oxford University Press.

Libby, R., \& Lewis, B. L. (1982). Human information processing research
in accounting: The state of the art 1982. \textit{Accounting,
Organisations, and Society, 7,} 231--285.

Kahneman, D., Slovic, P., \& Tversky, A. (1982). \textit{Judgments under
uncertainty: Heuristics and biases}. Cambridge: Cambridge University
Press.

Karelaia, N., \& Hogarth, R. M. (2008). Determinants of linear judgment:
a meta-analysis of lens studies. \textit{Psychological Bulletin,
134}(3), 404--426. 

Klayman, J. (1988). Cue discovery in probabilistic environments:
Uncertainty and experimentation. \textit{Journal of Experimental
Psychology: Learning, Memory, and Cognition}, \textit{14}(2), 317--330.

Mahan, R. P. (1988). Stress-induced strategy shifts toward intuitive
cognition: A cognitive continuum framework approach. \textit{Human
Performance}, \textit{7}(2), 85--118.

Marewski, J. N., \& Melhorn, K. (2011). Using the ACT-R architecture to
specify 39 quantitative process models of decision making.
\textit{Judgment and Decision Making}, \textit{6}(6), 439--519.

Meehl, P. E. (1954). \textit{Clinical versus statistical prediction: A
theoretical analysis and a review of the evidence}. Minneapolis:
University of Minnesota Press.

Miller, M. (1973). Interpersonal learning: Laboratory and field
investigations. In L. Rappoport \& D. Summers (Eds.), Human judgment
and social interaction (pp.\ 267--294). New York: Holt, Rinehart, \&
Winston.

Mosier, K. L., Skitka, L. J., Heers, S., \& Burdick,
M. (1998). Automation bias: Decision making and performance in
high-tech cockpits. \textit{The International Journal of Aviation
  Psychology, 8}(1), 47-63.

Mumpower, J. L. (2010). Disproportionality at the ``front
end'' of the child welfare services system: An analysis of
rates of referrals, ``hits,'' ``misses,'' and ``false alarms''.
\textit{Journal of Health and Human Services Administration},
\textit{33}(3), 364--405.

Mumpower, J., \& Adelman, L. (1980). The application of Brunswikian
methodology to policy formation. In K. R. Hammond, \& Wascoe, N. E.
(Eds.). \textit{Realizations of Brunswik’s representative design} (pp.
49--60). San Francisco: Jossey-Bass.

Mumpower, J. L. \& Hammond, K. R. (1974). Entangled task dimensions: An
impediment to interpersonal learning. \textit{Organizational Behavior
and Human Performance}, \textit{11}, 377--389.

Mumpower, J. L., \& McClelland, G. H. (2014). A signal detection theory
analysis of racial and ethnic disproportionality in the referral and
substantiation processes of the US child welfare services system.
\textit{Judgment and Decision Making}, \textit{9}(2), 114.

Mumpower, J. L., Nath, R., \& Stewart, T. R. (2002). Affirmative action,
duality of error, and the consequences of mispredicting the academic
performance of African American college applicants. \textit{Journal of
Policy Analysis and Management}, \textit{21}(1), 63--77.

Mumpower, J. L., Sheffield, J., Darling, T. A., \& Milter, R. G. (2004).
The accuracy of post-negotiation estimates of the other
negotiator's payoff. \textit{Group Decision and
Negotiation}, \textit{13}(3), 259--290. 

Mumpower, J. L., \& Stewart, T. R. (1996). Expert judgment and expert
disagreement. \textit{Thinking and Reasoning}, \textit{2}, 191--212. 

Mumpower, J. L., Veirs, V., \& Hammond, K. R. (1979). Scientific
information, social values, and policy formation: The application of
simulation models and judgment analysis to the Denver regional air
pollution problem. \textit{IEEE Transactions on Systems, Man, and
Cybernetics}, \textit{9}, 464--474.

Newton, J. R. (1965). Judgment and feedback in a quasi-clinical
situation. \textit{Journal of Personality and Social Psychology},
\textit{1}, 336--342. 

Neyman, J., \& Pearson, E. S. (1933). On the problem of the most
efficient tests of statistical hypotheses. \textit{Philosophical
Transactions of the Royal Society of London. Series A, 231}, 289--337. 

Phillips, L. (1982). Judgment and choice: An integration?
\textit{Contemporary Psychology}, \textit{27}(4), 312--313.

Poses, R. M., Cebul, R. D., Wigton, R. S., Centor, R. M., Collins, J.,
\& Fleischli, G. (1992). Controlled trial using computerized feedback
to improve physicians’ diagnostic judgments. \textit{Academic
Medicine}, \textit{67}, 345--347.

Reilly, B., \& Doherty, M. E. (1989). A note on the assessment of
self-insight in judgment research. \textit{Organizational Behavior and
Human Decision Processes, 44,} 123--131.

Reilly, B. A., \& Doherty, M. E. (1992). The assessment of self-insight
in judgment policies. \textit{Organizational Behavior and Human
Decision Processes, 53}, 285--309.

Rusou, Z., Zakay, D., \& Usher, M. (2013). Pitting intuitive and analytical thinking 
against each other: The case of transitivity. \textit{Psychonomic Bulletin and Review, 20},
608--614.

Seifert, M., \& Hadida, A. L. (2013). On the relative importance of
linear model and human judge(s) in combined forecasting.
\textit{Organizational Behavior and Human Decision Processes},
\textit{120}(1), 24--36.

Simon, H. A. (1957). \textit{Models of man: Social and rational}. New
York: Wiley.

Simon, H. A. (1991). \textit{Models of my life}. New York: Basic Books. 

Slovic, P., \& Lichtenstein, S. (1971). Comparison of Bayesian and
regression approaches to the study of information processing in
judgment. \textit{Organizational Behavior and Human Performance, 6}(6),
649--744.

Stewart, T. R. (1976). Components of correlation and extensions of the
lens model equation. \textit{Psychometrika}, \textit{41}(1), 101--120. 

Stewart, T. R. (1988). Judgment analysis. In B. Brehmer, \& C. R. B.
Joyce (Eds.), \textit{Human judgment: The SJT view} (pp.\ 41--74). New
York: North-Holland.

Stewart, T. R. (1990). A decomposition of the correlation coefficient
and its use in analysing forecasting skill. \textit{Weather and
Forecasting}, \textit{5}, 661--666. 

Stewart, T. R., \& Gelberd, L. (1976). Analysis of judgment policy: A
new approach for citizen participation in planning. \textit{Journal of
the American Institute of Planners}, \textit{42}, 33--41.

Stewart, T. R., \& Lusk, C.M. (1994). Seven components of judgmental
forecasting skill: Implications for research and the improvement of
forecasts. \textit{Journal of Forecasting}, \textit{13}, 575--599. 

Stewart, T. R., \& Mumpower, J. L. (2004). Detection and selection
decisions in the practice of screening mammography. \textit{Journal of
Policy Analysis and Management}, \textit{23}(4), 908--920.

Stewart, T. R., Roebber, P. J., \& Bosart, L. F. (1997). The importance
of the task in analyzing expert judgment. \textit{Organizational
Behavior and Human Decision Processes}, \textit{69}(3), 205--219.

Summers, D. A., Taliaferro, J. D., \& Fletcher, D. J. (1970). Subjective
vs objective description of judgment policy. \textit{Psychonomic
Science, 18,} 249--250.

\vfill
\break

Swets, J. A. (1992). The science of choosing the right decision
threshold in high stakes diagnoses. \textit{American Psychologist},
\textit{47}, 522--532.

Tanner, W. P., Jr., \& Swets, J. A. (1954). A decision making theory of
visual detection. \textit{Psychological Review}, \textit{61}, 401--409.

Tape, T. G., Kripal, J., \& Wigton, R. S. (1992). Comparing methods of
learning clinical prediction from case simulations. \textit{Medical
Decision Making}, \textit{12}, 213--221. 

Taylor, H. C., \& Russell, J. T. (1939). The relationship between
validity coefficients to the practical application of tests in
selection. \textit{Journal of Applied Psychology}, \textit{23},
565--578. 

Todd, F. J. (1954). A methodological study of clinical judgment.
Unpublished doctoral dissertation, University of Colorado.

Todd, F. J., \& Hammond, K. R. (1965). Differential feedback in two
multiple-cue probability learning tasks. \textit{Behavioral Science},
\textit{10}(4), 429--435. 

Tucker, L. R. (1964). A suggested alternative formulation in the
developments by Hursch, Hammond and Hursch, and by Hammond, Hursch and
Todd, \textit{Psychological Review, 71,} 528--530.

Tolman, E., \& Brunswik, E. (1935). The organism and the causal texture
of the environment. \textit{Psychological Review, 13}, 300--304.

Ullman, D. G., \& Doherty, M. E. (1984). Two determinants of the
diagnosis of hyperactivity: The child and the clinician. In M. Woraich
\& D. K. Routh (Eds.), \textit{Advances in behavioral pediatrics }(pp.
167--219). Greenwich, CT: JAI Press.

Wallace, H. A. (1923). What is in the corn judge’s mind? \textit{Journal
of the American Society of Agronomy, 42}, 43--77.

Way, B. B., Allen, M. H., Mumpower, J. L., Stewart, T. R., \& Banks, S.
M. (1998). Interrater agreement among psychiatrists in psychiatric
emergency assessments. \textit{American Journal of Psychiatry},
\textit{155}(10), 1423--1428. 

Wigton, R. S. (1988). Use of linear models to analyze physicians’
decisions. \textit{Medical Decision Making}, \textit{8}, 241--252.

Wigton, R. S. (1996). Social judgment theory and medical judgment.
\textit{Thinking \& Reasoning}, \textit{2}, 175--190.

\end{hangparas}
\end{document}
