2013

6th JDM workshop for young researchers – Report

The 6th JDM workshop for young researchers was held at the Max Planck Institute for Human Development in Berlin from July 17th to the 19th!

 

The workshop is a scientific event that combines the best features of a workshop and a conference. Its objective is to provide an opportunity for young researchers to network and discuss their research in a relaxed environment. The workshop was incepted in 2008 at the Max Planck Institute for Research on Collective Goods in Bonn, Germany, when a small group of PhD students decided that there was a need for young researchers to network and collaborate independently of senior researchers. Since then the workshop has constantly grown from year to year. The workshop has been held at the University of Mannheim (Germany), once more at the MPI in Bonn, and twice at the University of Basel (Switzerland).

Traditionally the organization team is recruited bottom-up from past participants to ensure continuity between years in layout and spirit. At the conclusion of last year’s workshop in Basel, Dirk Wulff and Nathaniel Phillips, two PhD students at the MPI for Human development, offered to host the workshop in Berlin. They thought the MPI for Human Development was particularly suited for this event as two of the four large research groups at the institute (the Center for Adaptive Behavior and Cognition, “ABC”, and the Center for Adaptive Rationality, “ARC”) conduct world-leading research on judgment and decision making rendering it one of the foremost institutions on JDM research in Europe.

Program

We received a total of 46 abstract submissions from pre-docs and post-docs coming from 13 different countries located on 4 continents – all records for the workshop. To ensure that all participants had opportunities to present and receive feedback, we instilled two changes to the workshop from previous years. First, we extended the workshop from the previous 2.5-day schedule to a full 3 days. Second, in addition to the traditional 15-minute talk and 15-minute discussion format, we created a new short presentation format with a 5-minute talk and 5-minute discussion. Using this new format, we were able to accept a total of 36 participants with everyone contributing an oral scientific presentation.

Participants in this year’s workshop had a diverse set of backgrounds, including Psychology, Organizational Behavior, Management, and Computer Science coming from a total of 24 different scientific intuitions around the world. The topics ranged from the very applied (e.g. Overconfidence and Entrepreneurial choice under Ambiguity by Anisa Shyti, HEC Paris Business School) to the very cognitive (Using computer mouse movements to parse the temporal dynamics of value-based-choices by Nicolette Sullivan, California Institute of Technology). Despite this wide range of research methods, there was however still a pronounced overlap in research questions as illustrated by the wordcloud from all accepted abstracts.

 

Keynote and Workshops

The workshop program also offered a keynote and two exciting workshops from experts in the field of judgment and decision making. For the keynote we were privileged to have Prof. Gerd Gigerenzer share his extensive experience as a researcher in a 90min advice session for young scientists titled “Publishing without perishing”. In his talk, Prof. Gigerenzer shared “5 ways to perish in publishing” (e.g.; “Ignore the reviewer’s points and interpret them as personal insults”), as well as 5 ways to increase your chances to publish (e.g.; “Be honest about the limitations of your research”). On the second day our invited speaker Prof. Gabriele Paolacci from the University of Rotterdam (Department of Marketing Management) demonstrated the benefits and risks of the Amazon Mechanical Turk for JDM research. Participants learned how the Amazon Mechanical Turk works as a general platform for crowdsourcing that offers a more diverse subject population and lower costs (time and money) than traditional participant pools. Besides, methodological questions and concerns of the participants were addressed in an extensive discussion. On the third day Dr. Mirjam Jenny and Dr. Stefan Herzog from the Max Planck Institute for Human Development gave a joint presentation on current political and methodological developments in the field of psychology in response to the recent cases of scientific misconduct and fraud. Their presentation provided a review of new developments, such as the Simonsohn’s p-curve method of detecting false positive results and encouraged participants to join the Open Science Framework to promote transparency in research. Finally, participants were given a hands-on demonstration of Bayesian statistical inference for standard statistical tests – including a simple web-page interface for conducting completely Bayesian “t-tests” based on Kruschke’s paper “Bayesian estimation supersedes the t-test.”

While the scientific presentations and workshops constituted a rich program during the day the workshop program also had a social program for participants to continue research discussions and share their experiences as young academics. On Wednesday and Friday we went out to restaurants in different areas of Berlin. On Thursday, we organized an official workshop BBQ for all participants and invited speakers together on the Max Planck Institute garden terrace.

The workshop was an overwhelming success and could not have been realized without the generous support from the European Association of Decision Making. Without the funding, we could not have invited external speakers, provided the workshop material, and offered coffee and refreshments to keep participants’ energy levels high throughout the workshop!

Future

Not only due to the success of this year the workshop’s continuation is already in planning. Next year Felix Henninger and Pascal Kieslich, who helped in the organization of this year’s workshop, will head the organization committee of the 7th JDM Workshop for young researchers at the University of Mannheim. We are looking forward to another great workshop featuring many interesting talks from international young JDM researchers and exciting lectures from experts in our field. Details will be announced this winter.

 

Organization committee

Dirk U. Wulff (chair)
Max Planck Institute for Human Development
Nathaniel D. Phillips
Max Planck Institute for HUman Development
Felix Henninger
University of Mannheim
Pascal J. Kieslich
University of Mannheim
Michael Schulte-Mecklenbeck
Max Planck Institute for Human Development

 

Small Group Meeting: Efficient Science – Report

On Friday, July 26th, 2013, a one-day small-group meeting entitled Efficient Science – Methodological Controversies in J/DM Research was held at the Max Planck Institute for Research on Collective Goods in Bonn. The meeting was jointly sponsored by the European Association of Decision Making and the Max Planck Society.

The small group meeting brought together researchers from the field of Judgment and Decision Making (J/DM) to reflect on the issues raised and the suggestions made in recent methodological debates in psychology. The specific aim was to assess the methodological situation in our field and, if necessary, to debate how our methodology can be improved so that science can be done more efficiently also in J/DM research. The problems and solutions discussed were grouped around the four main topics (1) Structural framework, (2) Design and data analysis, (3) Cumulative development of knowledge, and (4) Theory development.

The day started with keynote presentations by Michael Birnbaum (California State University, Fullerton) and Joseph Simmons (University of Pennsylvania). In his talks “Science of JDM as an Efficient Game of Mastermind” Michael Birnbaum demonstrated how to do efficient science by quickly narrowing down sets of hypotheses in as few steps as possible as in the game Mastermind. Besides showing similarities and differences, he exemplified how this approach can be realized in J/DM by testing critical properties of whole classes of theories. In the second keynote Joseph Simmons summarized recommendations to avoid a “false-positive psychology” and he provided further suggestions how to reach a positive true psychology. Both keynotes inspired constructive discussions and set the stage for the ten specific presentations that followed.

In the (1) Structural Framework section, Christoph Engel (MPI Collective Goods, Bonn) presented a game-theoretic analysis of the incentive structures in science which showed different possibilities how to influence individuals behavior in this public-bad dilemma for example by including reputation effects or uncertainty about when the strategic interaction ends. Frank Renkewitz (University of Erfurt), demonstrated structural problems for significance testing with large samples that follow from the fact that effects are likely to vary randomly determined by unknown factors. In the (2) Design and data analysis section, Erich Witte (University Hamburg) presented his suggestion to clearly differentiate between research in a phase of discovery and in a later phase of justification. Clintin Davis-Stober (University of Missouri) demonstrated that our estimations in experimental studies are often not better than chance and called for substantially higher powered studies to reduce the problem. Michael Schulte-Mecklenbeck (MPI for Human Development, Berlin) highlighted in his talk that even minor changes in research paradigms can influence results to some degree. In the (3) Cumulative development of knowledge section, Mark Brandt (Tilburg University) suggested that authors should prepare replication packages including material and data for published papers and make them publicly available on their webpage as standard of good scientific practice. Christoph Stahl (University of Cologne) discussed the advantages of a data-blind peer review process and informed that several journal plan to or have already included the submission category of “registered reports” as one possibility for publication. Mirjam Jenny (MPI for Human Development, Berlin) summarized recent developments in the methodology debate and presented data from a survey indicating that researchers also in the field of J/DM are open to change but that there is some skepticism concerning the introduction of strict rules. In the (4) Theory development section, Andreas Glöckner (University Göttingen) argued that one of the most fundamental problems to be solved in order to make science efficient in J/DM is to improve our theories to be better specified and to more thoroughly check the quality and the empirical content of theories to avoid the problem of theory accumulation. Klaus Fiedler (University Heidelberg) highlighted the importance of embedding theories in more general theoretical frameworks. He thereby argued against the mere focus on statistical hypothesis testing of isolated effects and in favor of searching for general explanation mechanisms and for more theory-driven thinking instead of focusing on alpha levels only.

Core insights and conclusion from the discussion were that (a) the generally problems identified in recent methodological debates also apply to J/DM (although perhaps partially to a smaller degree) and that the solutions suggested in other fields can basically be applied to J/DM as well. It was highlighted that (b) there is a need for educating young researchers that enter the field more intensely on methodology which should be implemented in summer schools. To achieve efficient science (c) there has to be more focus on general theoretical thinking, theory specification as well as theory testing and theory development. All researchers and institutions have (d) to contribute to change incentive structures so that high-quality research and good researchers are rewarded instead of rewarding pure quantity of output. This particularly concerns hiring committees, editors and reviewers of manuscripts and grant proposals. Finally, it became clear in the discussion that (e) all issues are intervened and that chances for success and advantages that could be potential gained are interdependent so that we have to move forward simultaneously in all respects to develop more efficient science.

Schedule – Program

9:20 – 10:20 Keynote: Michael Birnbaum: Science of JDM as an Efficient Game of Mastermind
10:35 – 11:35 Keynote: Joseph Simmons: Positive True Psychology
11:45 – 12:15 Christoph Engel: Scientific Dishonesty As a Public Bad
12:15 – 12:45 Frank Renkewitz: Random findings, alpha inflation, overrated relevance and generalisability: Consequences of the random variation of true effect sizes.
12:45 – 13:15 Erich H. Witte: Statistical inference techniques: Context of discovery and context of justification in empirical sciences – The long way of a research program
14:15 – 14:45 Clintin Davis-Stober: When are our experimental findings better than a guess?
14:45 – 15:15 Michael Schulte-Mecklenbeck: How replicable are process measures in decision making? The impact of subtle search costs
15:15 – 15:45 Mark J. Brandt: Advancing replicability and theory through replication recipes and replication packages
16:15 – 16:35 Christoph Stahl: Data-blind peer review
16:35 – 17:05 Mirjam Jenny: Psychologists are open to change, yet wary of rules
17:05 – 17:35 Andreas Glöckner: The empirical content of theories in judgment and decision making: Shortcomings and remedies
17:35 – 17:55 Klaus Fiedler: How Important is Statistical Hypothesis Testing for the Quality of Science?
Break
18:00 – 19:30 Discussion

 

Additionally to the two keynotes and ten presenters we had eleven guests without oral presentations, which made very valuable contributions in the discussion by asking critical questions, commenting on the suggestions provided and providing their opinions on various topics. Overall, 23 persons took part at this small group meeting which allowed for highly productive and intense discussions.

In the view of the organizers, the small group meeting achieved its goal to extend the methodological debate to the field of J/DM and to develop new suggestions for making science in our field more efficient. The contributions nicely complemented each other and participants got a differentiated overview over current debates, controversies and viewpoints. The insights developed at the meeting will be summarized in a joint multi-authored methodological article describing the core arguments discussed at the meeting. We thank the European Association for Decision Making (EADM) for the generous financial support of the meeting without which realizing the meeting would not have been possible.

Organization committee

Susann Fiedler, Max Planck Institute for Research on Collective Goods

Andreas Glöckner, University of Göttingen & Max Planck Institute for Research on Collective Goods

Klaus Fiedler, University of Heidelberg