“Cool stuff” vs. “incremental” in JDM research

Written by Ilana Ritov

In my previous column I argued for weighting intrinsic value over and above fit. One of the examples I gave involved an editorial decision about a paper. I suggested that the main goal of a journal should be to publish interesting papers. In this column I want to raise the question of what an interesting paper is or should be. This question is also linked to the increasingly criticized criteria for publication.

A few years ago, in a conference that brought together social psychologists and JDMers, I overheard a group of JDMers criticizing the “merely illustrative role” data seems to play in the mostly theoretical talks of the social psychologists, while the social psychologists, for their part made fun of the “found an effect” talks of the JDMers. This exchange may have been incidental, reflecting the views of the individual researchers more than the characteristics of their domains. However, I do find myself wondering whether the current JDM research is really becoming more and more the “found a (bizarre) effect” type?

In his recent paper in Perspectives on Psychological Science Paul Rozin suggests that in evaluating empirical papers, too much emphasis is given to faultless experimental design, at the expanse of the contribution of the research to our understanding of human behavior (http://bit.ly/cZf1Yz). When we find an effect, he says, “we are not rewarded for looking at the generality of the effect. Is it a fragile result of a carefully selected set of parameters? Or is it robust and operative across many situations and/or populations?” In the same paper Rozin expresses a critical view of our field’s overarching embrace of hypothesis-testing methodology. While I do not share his view regarding the role of hypothesis testing, I believe Rozin’s critique of the insufficient weight assigned to establishing the robustness of phenomena is on the mark. Reexamination of the highly conspicuous data on “the benefits of unconscious thinking” provides an exemplary exception, and a demonstration of the need for such research (Gonzales-Vallejo& Phillips http://bit.ly/ccWddc; Calvillo & Penzloza http://bit.ly/coUNeZ; for a meta-analysis see Acker http://bit.ly/cDy0zX).

I believe that the “found an effect” trend is at least partly driven by the policies of the top journals, most notably “Psychological Science”, the flagship journal of APS. As the editor of Psychological Science described it, he would like to publish “…the type of paper you would want to go down the hallway to psychologists who are not in your specialty area and say, ‘Look at this! This is really cool stuff’”. (http://bit.ly/aBgQdE).

But what is “cool stuff”? Granted, much of the literature is about exploring the causes and boundary conditions of effects. However, a complex analysis showing that a meaningful non-trivial effect occurs under some conditions but not under other conditions is likely to be dubbed “incremental”, certainly less “cool” than a surprising counterintuitive simple effect. While most of us, including myself have a taste for the counterintuitive, “cool stuff” that appeals to a large audience seems too likely to be the result of simplified overgeneralization or disputable analyses and could turn out to be more misleading than enlightening. Thus, the wish to appeal to a wider audience, and the considerable benefits that publishing in such journals entails, often results in a chase after this illusive “hit”.

Responding to similar sentiment of discontent with the way ideas and methodological issues are treated within JDM, Andreas Gloeckner and Ben Hilbig proposed a special issue of JDM journal on “Methodology of Judgment and Decision Making Research” (http://bit.ly/bbX7sV). The special issue they will edit will be asking whether it is sufficient to investigate effects, or do we need more complete models and ways of testing that would allow us to select between competing models. I hope this special issue will help elucidate some of the more fundamental methodological questions in our field, and will promote further the discussion of what the ingredients of an interesting and valuable paper are.