This is an open access article distributed under the terms of the Creative Commons
Attribution License (Yesterday, at a social event, one of us was chatting with another psychologist who remarked, “It is pretty incredible to be living in a period in which we get to observe the field undergo a historical transformation.” Their comment reflects a sentiment that many share—much has changed in how psychology is done and communicated during the last decade, perhaps more than in any other.
The motivation for change, and the largely grassroots activism that has advanced change, was accelerated around 2011 with events that highlighted potential weaknesses in the credibility of published evidence and claims, as well as the processes, practices, and reward systems that shape the production of evidence and claims. New approaches emerged to improve the credibility of research. Have the changes been for the better? Do the new approaches and behaviors address the right problems? What problems are being missed? What new problems are being created?
In 2021, ten years after the start of the “replication crisis” in psychology, we set out to collect a set of articles reflecting on the crisis and the ensuing “credibility revolution” or reform movement. The nine articles in this special topic provide a broad and varied perspective on this tumultuous decade.
Several of the submissions focus on ways in which the reform efforts have fallen short. These failures include, according to Beer et al. (2023, this volume): the reform movement has not been as aligned with efforts to improve diversity, equity, and inclusion in the field as it could be; it has narrowly focused on practices and researchers at research-intensive institutions; and it has focused on a narrow, homogenous subtype of research topics and designs. Another weakness of the reform movement, according to Wilson and Wixted (2023, this volume) is that most interpretations of replication projects, such as the Reproducibility Project: Psychology (Open Science Collaboration, 2015) are based on little more than intuition, while model-based approaches are more rigorous and challenge many of the intuition-based interpretations. Finally, one of the fundamental philosophical commitments (perhaps implicit) in the reform movement, the conceptualization of expertise, is misguided, according to Uygun Tunç and Tunç (2023, this volume). This has hampered progress, particularly in aligning incentives with the reform movement’s goals.
Other contributions focus on how we need to expand the scope of our self-reflection and reform efforts. In the early years of the crisis and revolution, the focus of the field’s attention was relatively narrow (mostly on p-hacking, false positives, and replication). However, the problems extend much farther. One source of error in our published research is problematic interpretive practices, and specifically the use of strategic ambiguity, according to Frankenhuis et al. (2023, this volume). Like many questionable research practices, these misleading interpretations can happen even when scientists act in good faith, and typically lead to overclaiming findings and their implications. Another overlooked source of potential errors and bias is the data from student projects, according to Ludwig et al. (2023, this volume). Their findings suggest that our pedagogical practices may encourage questionable and unethical research practices in student projects, which can easily make their way into published research. Finally, the perspective of participants is important to consider when evaluating potential threats to the validity of our research, according to Mason et al. (2023, this volume). Using qualitative evidence (from cognitive interviews and think-aloud protocols) they shed light on obstacles to straightforward interpretations of self-report data on common questionnaires.
A number of articles in this volume reflect on whether the reforms in our field have helped, and where they have fallen short. In the domain of first impressions research, Satchell et al. (2023, this volume) find that while much has improved (e.g., transparency, collaborative science, and norms around replication), there are still weak spots. In particular, the authors argue that most paradigms in this field are too asocial, and fail to reflect the rich, complex context of everyday life. Another area where we might expect improvement is in journal policies regarding replication research. Looking at social psychology journals’ policies with respect to replications, Torka et al. (2023, this volume) find that while the proportion of journal websites that explicitly welcome replication submissions has increased from 12% to 25% from 2015 to 2022, most journals still do not mention replication research on their websites. An important question for the reform movement to consider is whether the crisis, and the ensuing credibility revolution, have impacted public trust in science. Methner et al. (2023, this volume) found that lay people reported more trust in researchers if they learned about both the crisis and the reform movement, than if they learned about only the crisis (though the effect was quite small). This suggests that efforts to self-correct, when visible to the public, may help buffer any negative impact of visible failures and errors, but this may require sustained and comprehensive reforms (consistent with Schiavone & Vazire, 2023).
Although we are surely biased, as the editors who handled and accepted these articles for this special topic, we feel that the submissions themselves reflect the spirit of the revolution. They are reflective and critical, the empirical research is conducted rigorously and reported transparently, and the articles, we believe, make calibrated claims, aiming for accuracy over hype. Of course, there are just a handful of perspectives on the last decade in our field. However, they are emblematic of what may be the most important thing to emerge in the reform movement—an active and diverse research program on metascience.
The metascience movement draws on the long history of fields that examine the scientific process, and it adds a directional motive—how can we do better—and an entrepreneurial orientation—let’s try things to get better. The combination of an activist reform movement that is making change and a metascientific community that is offering solutions and evaluating their effectiveness is helping to overcome the inertia of the status quo, and the pull of system justification. When a dysfunctional system seems unchangeable, one way to resolve the dissonance of being a willing participant in it is to justify the existing system as the way it must be. Metascientists and reformers are demonstrating that the system is of our own making, and thereby subject to remaking.
The metascientific perspectives in this special topic capture both points of pride as well as mistakes or missed opportunities, and provide some fodder for reflection. Moreover, as the reform movement has spread across scholarly disciplines, the experience of psychologists and their expertise in human behavior are tremendous assets to help continue on the path towards self-improvement and increasing credibility, not just for psychology but for all domains of scholarly inquiry.