Lifetime achievement award to Robin M. Hogarth


Reflections on judgmental ability
Robin M. Hogarth

Universitat Pompeu Fabra, Barcelona
Speech prepared for EADM’s lifetime achievement award (August 20, 2023).

Lifetime achievement award lecture

First, let me say how honored I am to receive this award for lifetime contributions. However, although I am the person who receives the award, I would not be in this position if I hadn’t – over the years — had such a remarkable group of colleagues and students with whom to collaborate. So, the award is due to the many colleagues and students who illuminated my professional career, and I thank them for sharing my journey with them.

Over the years, the SPUDM conferences have provided a means to meet colleagues and to share findings and ideas and it has always been my favorite, especially as it takes place in an international context. The first SPUDM conference that I attended was 50 years ago, in Rome in 1973 and I think it is appropriate to reflect on that event today.

Rome 1973 was hosted by the legendary Bruno de Finetti who had set a goal for SPUDM in Hamburg in 1969 by stating that

“The true….subjective probability….problem consists in the investigations concerning the ways in which probabilities are assessed by more or less educated people and the way in which such abilities may be improved. This seems to me the field in which the cooperation between all specialists concerned is most wanted, and that is particularly true for the expected contributions from psychologists.”

I will come back to this quote at the end of this talk.

Accommodation in Rome was arranged in student dormitories, which were modest, to say the least, and the weather was hot and damp. There were about 25 participants and the sessions all took place in the same lecture hall of the University of Rome. There was just one stream of talks.

For me, it was an exciting experience – meeting people whose work I had read and admired, getting to know other researchers, and starting to make friendships with several people with whom I have continued to interact over the years. I came away from the conference in a buzz, with an excited feeling that decision research was the future…….
And so it was!

For the first paper presented at the conference was the famous 1974 Science paper by Amos Tversky and Daniel Kahneman on “heuristics and biases.” After papers, there were questions and discussion. And I remember well the gist of the first question posed to Tversky and Kahneman. It was “If you think that people make mistakes like your experimental subjects do, how do you exclude the possibility that you yourselves are not making mistakes in asserting this?” Tversky briefly acknowledged the pertinence of the question and Kahneman replied that they had followed all the scientific recommendations for testing hypotheses for different results.

Now, you may ask why I recount this incident. Well, first it was I who asked the question, and second it raises the issue that I want to address in this brief talk, namely “How good is human judgment?” What can we say 50 years after 1973?

Let me start by asking what we know.
• Humans have limited memory and information processing capacity both of which emphasize the importance of attention (Simon).
• Importance of task characteristics and especially feedback – differences between kind and wicked learning environments.
• Two modes of thought…tacit (System 1) and deliberate (System 2).
• On the other hand, humans have remarkable abilities for recognition (Goldstein & Gigerenzer) and a capacity to recall the frequency of past experiences (Hasher & Zacks).
As an example of the latter, imagine that somebody asks you how often you have been to the cinema in the last six months. The important result is that most people experience little difficulty in answering the question even though they had made no prior effort to recall the frequency of cinema visits.
• Much of human learning is tacit (Reber).

Major concern: How good (overall) may not be the right question – perhaps it is better to ask how good ability needs to be and therefore when you can expect to see effective or ineffective judgments.

In terms of task characteristics, one of the limitations of most research is that it is conducted within what I call a discrete model of judgments/decisions. That is, participants make a judgment, and this is subsequently seen to be correct or not. It is like firing at a target and then seeing how accurate the result is. (The target analogy is, incidentally, also used by Daniel Kahneman and his co-workers in the much-publicized book Noise).

A critical feature of this model is that at an individual level participants receive no feedback. They make their judgments and experimenters then determine how good they are. And yet this makes a difference. As Lejarraga and Hertwig (2022) document in a recent comprehensive review, there are important differences between inferences made in situations where people do or do not receive feedback.

In Hogarth and Soyer (2011), for example, we demonstrated how estimates were dependent on whether people answered probabilistic questions with or without feedback and showed that experience was superior to description.

Percentages of Correct Answers to Inferential Problems by Experimental Conditions

In this paper, we investigated how participants’ responses would vary depending on whether they answered a description or experienced a sample of observations. There were 2 groups of participants, “sophisticated” and “naïve”, the former being paid students and the latter volunteers from acquaintances. Each respondent saw 2 versions of the same 7 problems and we saw how responses varied when they relied on analysis or experience. (I skip the experimental design explanation.)

The table shows the responses to 2 of the 7 problems. As can be seen, responses based on experience are almost perfect for the Bayesian updating problem (98%) and in the majority for the birthday problem (60%).

Moving on from these results on differences between description and experience, I would like to make the claim that many judgments can be better described as being of a continuous as opposed to a discrete nature. Let me provide some examples. Imagine that you want to leave the lecture hall and need to select a route from your seat to the door. You could treat this as a discrete task that would involve the selection and use of a particular path that you would follow. However, it is more likely that on leaving your seat you would make a rough estimate of the direction of the door and update this judgment as you advance to the exit.

Indeed, the judgmental processes involved in walking, driving, and even conversations follow a model of a series of rough judgments made across time as a function of the feedback received on actions. It reminds one of the old joke about a Japanese airplane pilot who ditched his plane some 100 meters from the runway and into the sea at the San Francisco airport. When asked why he missed the runway, his reply was clear: “Considering that I flew from Tokyo, missing the runway at SFO by a few hundred meters was not bad.”

In short, in continuous judgmental tasks, people make a series of judgmental adjustments that allows them to reach their goal – the key point being that they are only really committed by the last adjustment that they make.

We can clarify these ideas by using the target analogy where judgment can be likened to aiming at a target, and choice represents the selection of a particular trajectory (i.e., shooting). For example, consider Figure 1 and imagine that you are at point A. The precise target is the point D, and the line BC shows the amount of allowable error.

As concrete examples, imagine predicting future values of an economic variable (e.g., sales, gross national product) or level of success in a job or graduate school. In both cases, choice is represented by the selection of a particular level of the variable.

Now consider the probability of hitting the target in Figure 1 at random (i.e., in the absence of predictive ability). Provided that you are “pointed” in the appropriate direction — indicated in Figure 1 by drawing EF parallel to BC—the probability of hitting the target BC at random in a discrete incident is equal to the ratio of the angle a (i.e., BAC) to 180°. However, a is a function of BC (size of target) to AD (distance from target). Therefore, the probability of hitting the target without predictive ability is an increasing function of BC/AD (i.e., size of target/distance from target).

For example, when the ratio of target to distance is .10, the probability of hitting the target at random in a discrete incident is .032. However, starting from the same point, a continuous process represents a much easier task. By simply moving toward the target and checking periodically on direction, the judge can transform the initial probability of .032 to almost a certainty, without exercising much predictive ability.

As a concrete example, imagine a sensitive interview where one would not want to pose a direct question before acquiring a more complete appreciation of the situation. This can be illustrated by comparing Figures 1 and 2.

Consider the effects of recognizing that the positions of the cues L and M provide information; in particular, that an action to the right of L, and to the left of M, is appropriate. In this case, the probability of hitting the target at random conditional on this knowledge is a divided by the angle LAM. Thus, the joint validity of the cues L and M can be measured by the
differential predictive information they provide over the base rate.

Now consider Figure 2 in which the person has moved from position A to position
X (i.e., after receiving feedback from initial directing questions). This has two effects: first, the base rate of hitting the target has increased, since angle BXC is greater than angle BAG; second, the predictive validities of L and M have decreased relative to Figure 1 (angle LXM is greater than angle LAM). Of course, it is unclear that as a person moves toward a target he or she would continue to rely on the same cues. In an interview, for instance, one formulates new questions due to prior feedback.

How appropriate is the continuous model? It certainly applies to many simple and even complex motor tasks (e.g., walking through a door or skiing), as well as more cognitive tasks that induce feedback through actions. For example, monitoring the performance of a trainee in a new job provides intermediate feedback that can be used to modify predictions and/or actions. More importantly, the continuous model highlights two critical dimensions of judgmental achievement: (a) the degree of commitment implied by actions and (b) that the availability and interpretation of feedback are often more important than predictive ability per se.

This all leads me to two questions:
Question 1: What proportion of our decisions is made in continuous mode?
Question 2: How aware are we of feedback?

Unfortunately I cannot answer the first question. It is hard to estimate the proportion of decisions that are taken in continuous mode. But, on reflection, it must be quite large.

To illuminate the second question, several years ago I conducted a study of decisions taken by business managers and undergraduate students using the experience sampling method (ESM). The task faced by the participants involved responding to SMS messages by describing the latest decision they had made together with information about feedback. Results showed that participants received or expected to receive feedback for only 60% of their decisions. In other words, much of their decision-making activities involved no feedback, of which they were consciously aware. This therefore suggests, inter alia, that tacit processes play a large role in the interpretation of feedback.

From a research perspective, what are the effects of adopting a continuous as opposed to a discrete perspective?

The continuous perspective is not a maximizing paradigm, and it opens the question as to what rationales people invoke when making decisions in this way.

One view is that people can be thought of as managing a portfolio of their “assets” and the rationale behind their decisions is “to stay in the game.”

Evidence supporting this hypothesis was provided by Ronen (1973) in a simple gambling task. Ronen presented participants with the choice between two mutually exclusive actions characterized by identical (positive) payoffs and probabilities of success. The payoffs, however, depended on the outcomes of two sequential events where the “paths” to success differed in that one had a higher first-stage probability of success but a lower second-stage probability. These probabilities were known to the subjects.

Since the expected utilities of the actions were equal, people should have been indifferent to choosing between them. However, they were not. Subjects largely favored the action with the greater first-stage probability.

Why? A plausible hypothesis is that people act with the immediate goal of reaching a better position from which to reach the ultimate outcome. Moreover, if people are accustomed to environments changing over time, less confidence should be given to the second stage probability. Thus, staying in the game as long as possible favors the action with the larger first-stage probability of success.

It also follows, of course, that if losses are at stake, people will seek to withdraw from the situation as soon as possible. And this was the case in the experiments.

Reflecting on discrete-continuous issues, I was led to imagine that the essence of decision making was how people manage portfolios of assets across time. This task can be captured by imagining attempts to cross a “foggy mine field.” There are multiple paths that can be taken, mines that can and cannot be detected, and rewards that can be achieved with different chances of success.

Indeed, this is like life itself. We all manage portfolios that include financial assets, professional performance, health, and happiness, and must deal with outcomes in all areas continuously. With some colleagues, we went as far as developing an experimental version of this paradigm which we named “the life game.” We had hoped to investigate how people would react to different situations in computer simulations of the life game. However, our experimental participants rated the task as “too boring” and, and this combined with some other issues led us to abandon the project.

And so where do we now stand with respect to the assessment of human decision-making ability?

First, there are clearly situations where people are biased according to the discrete paradigm. But there is some variation between tasks.
Second, the importance of task characteristics is reinforced by considering the effects of the continuous paradigm. This shows that people`s judgments can be trusted in many situations. How much, however, is not clear and needs more research.

All the above issues suggest some implications for future research.
First, are there ways to improve forecasts?
Transform discrete tasks into a continuous format.
Evidence from “superforecasters”(delaying responses and updating frequently)
City planning
Managing interviews
Other tasks…..

Second, can we use the continuous model to improve understanding of diagnostic tasks?

Third, the work probably relates to a recently proposed framework for decisions from experience (Olschewski et al., 2023).

Finally, what can we say about the validity of human judgment and respond to de Finetti’s challenge that I quoted at the beginning of this talk?

• Cognitive biases in probabilistic tasks are real but can be overcome by specific training, e.g., systematically using base rates, or averaging judgments to eliminate noise.
• We cannot make an overall assessment because it is not clear what the population of judgments is. There are multiple judgments where people use task characteristics to make judgment feasible and we don’t know enough about the universe of these judgments.
• The human judgmental system is extraordinarily adaptable.

Workshop: Navigating Clinical Uncertainty

Navigating Clinical Uncertainty

10th European Workshop on Clinical Reasoning and Decision Making &

COGITA: Gut Feelings in General Practice

March 26th – March 28th , 2015

Marburg (Germany)

Dear colleagues,

we gladly announce the 10th workshop on Clinical Decision Making and Diagnostic Reasoning, this time in collaboration with the COGITA-group, a European network of researchers focusing on the role of gut feelings in the diagnostic reasoning of general practitioners. This conference will be hosted by the Departments of General Practice and Clinical Psychology at the University of Marburg (Germany). The aim of this interdisciplinary workshop is to bring together researchers from decision and clinical science as well as clinicians to share their ideas, their research and their experience. The workshop is sponsored by the European Association for Decision Making and attendance is free.

For more information see

We are looking forward to seeing you all in Marburg!


The Scientific Committee:

Norbert Donner-Banzhoff (Marburg/D), Laurence Claes (Leuven/B), York Hagmayer (Göttingen/D), Erik Stolper (Maastricht/NL), Winfried Rief (Marburg/D), Cilia Witteman (Nijmegen/NL)

Registration open for SPUDM 24

The European Association for Decision Making invites you to attend its next biannual 24th Subjective Probability, Utility, and Decision Making Conference (SPUDM 24), which will be held at IESE Business School – University of Navarra in Barcelona, Spain, on August 18-22, 2013.

We are pleased to announce that the registration for the conference is already open and available at:

The conference will feature the following invited speakers:

*        Timothy D. Wilson, University of Virginia, USA

*        Colin F. Camerer, California Institute of Technology, USA

*        Robin Hogarth, Universitat Pompeu Fabra, Spain

*        Ralph Hertwig, Max Planck Institute for Human Development, Berlin, Germany

Attending this meeting will also be an opportunity to discover Barcelona, one of the most unique and architecturally distinctive cities of the world. Barcelona is the capital of Spain’s Catalan region, which has produced a number of the world’s most prominent artists including Pablo Picasso and Salvador Dalí. The architect Antoni Gaudí also left his indelible mark on the city through a number of remarkable buildings such as La Sagrada Familia, La Pedrera, and La Casa Batlló.

We look forward to seeing and welcoming you in Barcelona!

The local organizing committee:

Elena Reustkaja (IESE, Spain); Mario Capizzani (IESE, Spain), Franz Heukamp (IESE, Spain), and Robin Hogarth (UPF, Spain).

SPUDM 2013 Call for papers and submissions

The European Association for Decision Making invites submissions for its 2013 Biennial SPUDM24 conference to be held at IESE Business School, Barcelona, Spain from Sunday, the 18th till Thursday, the 22nd of August 2013.

Submissions of paper abstracts, poster abstracts, and proposals for workshops are invited on any topic in basic and applied judgment and decision making research.

Deadline for all submissions is March 8, 2013.

The organizing committee is pleased to announce that the conference will feature the following invited speakers:

  • Timothy D. Wilson, University of Virginia, USA
  • Colin F. Camerer, California Institute of Technology, USA
  • Robin Hogarth, Universitat Pompeu Fabra, Spain
  • Ralph Hertwig, Max Planck Institute for Human Development, Berlin, Germany

Attending this meeting will also be an opportunity to discover Barcelona, one of the most unique and architecturally distinctive cities of the world. Barcelona is the capital of Spain’s Catalan region, which has produced a number of the world’s most prominent artists including Pablo Picasso and Salvador Dalí. The architect Antoni Gaudí also left his indelible mark on the city through a number of remarkable buildings such as La Sagrada Familia, La Pedrera, and La Casa Batlló.

For call for papers and submissions, please, visit

We look forward to seeing you and to welcoming you to Barcelona!

The local organizing committee:

Elena Reustkaja, Mario Capizzani, Franz Heukamp, and Robin Hogarth

SPUDM 2013

EADM invites submissions for its

2013 Biennial SPUDM24 conference

SPUDM24 will be held at IESE Business School, Barcelona, Spain from 18-22 August 2013. The organizing committee is pleased to announce that the conference will feature the following invited speakers: Timothy D. Wilson, Colin F. Camerer, Robin Hogarth, and Ralph Hertwig. For call for papers and submissions, please visit the SPUDM24 website.

EADM members join the JDM journal supervisory committee

The journal Judgment and Decision Making is now the joint journal of the Society for Judgment and Decision Making (SJDM) and the European Association for Decision Making. This means that it will be supervised by a committee consisting of two members representing each organization, appointed by the president of each. The supervisory committee has complete, ultimate control over all matters pertaining to the journal, although it is assumed that the editor (Jon Baron) will make day-to-day decisions and will consult the committee only for fairly major issues (such as changes in policy), or when he needs advice. SJDM has decided that it’s executive board must approve the appointment of the next editor, after the committee makes a recommendation. The committee will consist of:

  • Derek Koehler, University of Waterloo (SJDM)
  • Gary McClelland, University of Colorado (SJDM)
  • Cilia Witteman, Radboud University (EADM)
  • Nicolao Bonini, University of Trento (EADM)

Judgment and Decision Making as a Skill

Recent publication from our member Mandeep Dhami

This book presents a comprehensive review of emerging theories and research on the dynamic nature of human judgment and decision making (JDM). Leading researchers in the fields of JDM, cognitive development, human learning and neuroscience discuss short-term and long-term changes in JDM skills.

The authors consider how such skills increase and decline on a developmental scale in children, adolescents and the elderly; how they may be learned; and how JDM skills can be improved and aided. In addition, beyond these behavioral approaches to understanding JDM as a skill, the book provides fascinating new insights from recent evolutionary and neuropsychological approaches.

The authors identify opportunities for future research on the acquisition and changing nature of JDM. In a concluding chapter, eminent past presidents of the Society for Judgment and Decision Making provide personal reflections and perspectives on the notion of JDM as a dynamic skill.

Dhami, M. K., Schlottmann, A., & Waldmann, M. (2011). Judgment and Decision Making as a Skill. Cambridge, UK: Cambridge University Press.

Review of Perspectives on Thinking, Judging, and Decision Making

Perspectives on Thinking, Judging, and Decision Making is a collection of cutting edge research that offers an approachable and up to date landscape of the judgment and decision making area.

The book is great: Read it! … Not convinced yet? Well, here’s more: 

This Festschrift is a great source of knowledge and inspiration for anyone interested in the way we think, judge, reason and make decisions. We would not expect less of a tribute to Karl-Halvor Teigen, who blazed many trails in the social-cognitive sciences and whose scientific and human contribution stimulates much current research.

At first the book may appear as a collection of (very interesting) papers that presents innovative theoretical or empirical findings at both a basic and an applied level (e.g., chap. 12 looks at the determinant of stock market anomalies). But the book is more than a collection of (very good) chapters. The chapters share a vision of human cognition and an objective.

The chapters approach human cognition as an enigma, wondering for example, “how do we think?” and “how could we think better?” The questions examined stem from surprising details, peculiar phenomena and pitfalls of reality observed in our lives. All these questions arise from daily life and have thus pervasive implications. This bottom-up approach is the backbone of the book. This approach prompts us – researchers, researchers-to-be or simply inquisitive minds – to be inspired by what surrounds us and to root our reflections about people and their mental processes in people’s own (subjective) reality. The bottom-up approach helps to question pre-established views and encourages research creativity and, in doing so, provides a refreshing perspective on the challenges emerging in thinking, judgment and decision making sciences.

Of course, the book does not only raise questions; international experts provide insightful answers. Drawn from various disciplines, the authors tackle human cognition from different angles and provide the reader with varying views, methods and reflections. Perspectives on thinking, judging, and decision making examines different factors that contribute to enhance or hinder human cognition; such as when and where information is provided, to what is it compared, and who is around us. The multiplicity of constructs under focus and the use of various lenses contribute to a meaningful picture of the marvels and snares of human cognition. The book shows that our judgments and decisions are often based on irrelevant cues, are occasionally normative, but above all, are always fascinating.

The book testifies to the excellent research that is conducted in judgment and decision making and to the concerns of this field for progress in knowledge and lives. Some chapters offer valuable theoretical developments (e.g., chap. 1 develops a Taxonomy of uncertainty) whereas others provide new insights on past findings and enable us to rediscover previous work in a new light (e.g., chap. 21 walks us back to the work of Descartes on (un)conscious and emotion). The book also includes chapters that help us to re-think the role of accepted findings (e.g., chap. 9 revisits the role of intuition in conjunction fallacy) and to integrate new findings from one field into a known framework from another (e.g., chap. 22 integrates findings on ethnics bias in the dual system framework).

Beyond being a source of academic knowledge, this book is also highly entertaining. Personally, what contributed most to my reading experience, and what makes the book essentially different from other books on JDM, is the inclusion of many comic anecdotes and fond moments that the authors shared with Karl-Halvor. The book testifies to the fun and friendship fostered during scientists’ careers. I take the occasion of this review to thank the editors for their efforts, time and commitment putting this book together, it was really worth it!

Perspectives on Thinking, Judging, and Decision making makes a great contribution to advancing our understanding of how we think, judge and decide and of the factors that can help us do better. The book is an excellent resource for researchers and an appealing introduction to the field for beginners.

Marie Juanchich

EADM JDM Summer School 2012

The Psychology Department at the University of Essex, in association with the European Association of Decision Making, played host to the first Judgment and Decision Making Summer School at the end of August 2012. Current, and recently completed, PhD students from across Europe, researching any area in judgment and decision making, attended an 8 day programme put together by Dr Tim Rakow. The guest lecturers who attended were Dr. Glöckner (Max Planck Institute for Research on Collective Goods), Prof. Neil Stewart (University of Warwick) and Dr. Mandeep Dhami (University of Surrey).

The program offered an eclectic view on the field of judgment and decision making, including methodological advances using eye-tracking devices, modelling of risky choice, introduction to Bayesian hypotheses testing, as well as real world applications in the legal and criminal fields. In addition to the lectures, the summer school also had practical workshops in order to give attendees a ‘hands-on’ experience with new equipment and software.

The computer workshops offered an introduction to Bayesian analysis and model fitting using R and Matlab. Other workshops included a tutorial on setting up experiments and analysing eye-tracking data, as well as interactive workshop devoted to the critical review of the guidelines currently employed in the UK’s criminal justice system. The lecturers organised several sessions to discuss issues affecting new researchers such as the challenges associated with the peer-review process and carrying out research in an applied setting.

The entire course was also complemented by excellent keynote speakers including Prof. Peter Ayton (City University London), Prof. Nigel Harvey (University College London), Dr. Mitch Callan (University of Essex), Dr. William Matthews (University of Essex) and Dr. Nick Sevdalis (Imperial College London), who delivered inspiring talks about their research.

The summer school provided an excellent opportunity for new researchers to interact and get to know others who share similar interests in judgment and decision making through a medium of poster presentations, shared meals and exciting social events (including a trip to the Paralympics). We are confident that such an experience aided the generation of new exciting research ideas, as well as the prospects of new collaborations.