Hits and Misses in the Last Decade of Open Science: Researchers From Different Subfields and Career Stages Offer Personal Reflections and Suggestions

Is there a reason to be optimistic that psychology is doing all it can to correct questionable research practices, bolster replicability, and regain credibility? The answer to this question depends on whom you ask. While many high-profile researchers debate the utility of various best practices, less attention has been paid to systematically considering how each practice affects researchers from different subdisciplines, career stages, identities, and institutional contexts where their research is conducted. In this article, we give insight into how a recently-held conversation on the impact and future of the Open Science movement might play out between three early- and later-stage career researchers who focus on various subdisciplines (diversity science, relationship science, and social neuroscience and social cognition) and conduct their research at universities that place differing levels of emphasis on research.

The genesis of this article was as follows: Dr. Beer first broached this topic with Dr. Eastwick, and then they both invited Dr. Goh to add some additional perspective. All three authors work in academia in the United States, and they agreed that their collective expertise put them in a position to pose and answer questions at the intersection of Open Science and (a) Diversity, Equity, and Inclusion (DEI), (b) data sharing, and (c) preregistration. In order to ensure that each researcher maintains their voice in this collaborative conversation, the article indicates sections that represent thoughts from specific authors (e.g., avoiding the need to agree on an opinion that caters to the median of the group: Forscher et al., 2020).

Dr. Jin Goh received his PhD in 2017 and is an assistant professor at Colby College, a small liberal arts college; he will reflect on the changes in his subfield of diversity science, since his participation as an undergraduate student in the research consortium behind a massive replication effort of psychological findings (Open Science Research Consortium, 2015). Dr. Goh’s perspectives are shaped by his relatively early career stage and his identity as a gay Asian American man. While his race and sexual orientation may at times marginalize him within social-personality psychology, his gender and able-body privilege him to navigate and speak up comfortably in the field. Within diversity science, he uses both experimental and correlational designs to study how identities and institutions influence intergroup perceptions.

Dr. Paul Eastwick received his PhD in 2009 and is a professor at the University of California, Davis; he will reflect on the broader context of changes in the field of relationship science. Dr. Eastwick’s views of the Open Science movement were shaped by the fact that he was trained in both experimental designs and the use of multilevel modeling in dyadic and longitudinal datasets; these different knowledge bases led him to be attuned to the way that solutions might work well for one context and poorly for another. Also, at about the time that the Open Science movement was gaining prominence in Psychology around 2014, he was (nearly) tenured at a research-focused institution, so he had the luxury of having the time to adapt to the new norms while also being in a position to critique the movement—and the backlash to the movement—without incurring much personal risk.

Dr. Jennifer Beer received her PhD in 2002 and is a professor at the University of Texas at Austin; she will reflect on the progress made in both psychology and neuroscience since she participated in the same research consortium as Dr. Goh (e.g., Open Science Research Consortium, 2015). Dr. Beer’s perspective has been shaped by being a member of an underrepresented group in one of her fields, working with convenience and non-convenience samples using behavioral manipulation and neuroimaging techniques, as well as training and working at research-focused, public universities. All three authors were trained and have worked in the United States and are therefore mostly familiar with the open science movement in that context.

Our stances and opinions range from optimistic pessimism to pessimistic optimism. From all of our vantage points, we would not expect every problem to have been solved by now. And there have been many successes in the shift toward practices that are aimed at reducing questionable research practices and bolstering replicability. However, one emergent theme is that the practice(s) which are most impactful can differ by subfield. For example, the call for larger samples is great in principle but is not easily attainable by many researchers. And some areas have seen the greatest benefits from the practice of pre-registration while others have made more strides because of the recommendation to facilitate data sharing.

We note areas of convergence and divergence about the ways in which the changes in best practices have been successful and where they could use more development. We begin by describing the intersection of the open science movement and DEI issues. We next discuss the successes and failures around the shift toward greater data availability and pre-registration. We conclude with six specific recommendations to ensure that the field’s ability to correct itself includes all interested researchers and knocks down the current infrastructure which paradoxically impedes scientific progress.

How Do You Think Open Science and Diversity, Equity, and Inclusivity (DEI) Intersect?

(From PE) There is considerable potential for the open science movement to impact diversity, equity, and inclusion (DEI) in a positive way, but some goals of the movement are more obviously connected to DEI efforts than others. Consider that around 2011–2012, part of the original impetus for the open science movement was rooted in frustrations expressed by some scholars directed at other scholars whose work failed to replicate (see Nelson et al., 2018, for examples). If the movement were to revolve around grievances like these—the desire for some scholars to correct the sins of the past through exposé and criticism—such practices might improve the credibility of the existing scientific record, but it is not clear that such practices would help DEI efforts. Indeed, they might harm DEI by maintaining a climate of competition and reinforcing the notion that there are scientific “winners” and “losers” (Murphy et al., 2020).

An alternative framework is to view open science as synonymous with inclusive science (Ledgerwood et al., 2022). According to this view, the ultimate goal of the open science movement would be (a) to enable people of all backgrounds/levels of seniority to be full participants in the scientific process, (b) to reduce the advantages bestowed upon insiders and senior scholars in academic culture, and (c) to make the products of science (e.g., papers, data, code) more transparent and available for all consumers. Many of the organizations and structures that have grown out of the open science movement (e.g., the Society for the Improvement of Psychological Science, the Facebook group “PsychMAP,” The Center for Open Science) espouse values that are closer to the goals of inclusive science than the goals of a replicability-bolstering science. Nevertheless, there are likely to be continuing tensions when these goals come into conflict, such as when senior (usually male) scholars deride the presumed replicability of the work of junior scholars on unmoderated platforms like Twitter (Whitaker & Guest, 2020).

(From JG) I entered graduate school in the summer of 2012, which I spent excitedly coding articles for the Open Science Research Consortium replicability project. Discussing Bem (2011) in my undergraduate lab invigorated me to enter social psychology with an eye toward open science. I soon discovered my zeal was not widely shared in graduate school; at times I butted heads with senior faculty and graduate students. Reflecting back on the early days of the Open Science movement, my vantage point then is different from PE and even myself right now. As a graduate student then, it felt as though I was watching metaphorical giants hurling rocks (or words) at one another and everyone else had to avoid becoming casualties in their fights (or debates). Conferences, social media, and departmental meetings felt like dangerous waters because it was difficult to know who might be offended by your opinions on open science. I could not help but admire the graduate students who were willing to challenge established figures in the field.

So how did these debates impact DEI within the Open Science movement? Researchers (primarily tenured, White, and/or male researchers) at “R1” institutions dominate academia and its various leadership positions such as search committee chairs, journal editors, and professional society leaders (Buchanan et al., 2021; Roberts et al., 2020; Syed & Kathawalla, 2022). As such, their perspectives on open science were likely skewed by their experiences and positions. Yet being in positions of power, their skewed perspectives shaped the norms and journal practices that the rest of us had to follow. At the time, social-personality psychology, and the discussion of its future, was seemingly reserved for a particular demographic and everyone else was a spectator.

This situation only worsened when researchers (often from marginalized backgrounds) tried to discuss the role of DEI and received criticism for it. Some argued that open science was independent of biases because scholars were merely discussing ideas and practices and such debates should be divorced from any researcher’s identity and position. “Why bring up race or gender,” some wondered. From my perspective, this colorblind approach negated careful considerations of DEI in the early days of the movement (Ledgerwood, 2017; Srivastava, 2014).

The early Open Science movement championed itself as a free discussion of ideas and practices that all could participate, and in many ways, that is true and it is increasingly normative to discuss DEI. However, I could not help but wonder who is socialized to speak freely and comfortably of their ideas without worrying about the precariousness of their status within academia and society broadly?

(From JB) I agree with all of the points that Dr. Goh and Dr. Eastwick raise. I will also say that I have been disheartened by Twitter arguments and “behind the scenes” failures to support open science which have affected trainees’ interest in continuing in science for their career. One theme I’ve been struck by after recent conversations with students is a feeling of “doing everything right” in terms of adopting pre-registration to be transparent yet still end up being accused by editors or reviewers of QRPs which sometimes leads to the feeling that the field does not really care about improving science. Instead, we may have just exchanged one set of bullies or gatekeepers for another. And they want nothing to do with science as a result. In light of feelings like this, I have to conclude that despite some benefits, it is a huge shame if open science is driving some people away from the scientific community.

How Easy and Effective Has It Been to Adopt Data Sharing Practices Within Your Subfield?

The call for larger samples has been met with new infrastructure to facilitate collaboration (from JG).

We are all in agreement that the renewed emphasis on large samples is better for our science in terms of statistical power and effect size estimation. However, the emphasis on large samples has the potential to exclude researchers working in contexts with relatively less research support but has been met with new infrastructure to facilitate collaboration among scholars.

Instead of basing conclusions on 50 people for a factorial ANOVA, we are now seeing 500+ people in multi-study articles to achieve statistical power (Giner-Sorolla, 2018). Even in short journal formats, a perusal of published articles in 2022 showed that the majority of articles had more than one study, with total sample sizes that often exceed 500. These improvements have permitted greater statistical power and more accurate inferences, but there are limits and shortcomings. First, we are becoming overly reliant on convenient and cheap (exploitive) sampling via MTurk and other online recruitment platforms (Buhrmester et al., 2018). Due to the online nature of such studies, we may be moving farther away from rich behavioral studies that distinguish social-personality psychology from other subfields (Doliński, 2018). As an early career faculty, tenure expectations demand publications in top journals that often require multi-study packages. Many researchers (myself included) would rather rely on convenience sampling than time-intensive studies that may not pay off. The convenience of online sampling as well as the demand for large samples are pushing us to become even more of a “science of self-reports and finger movements” (Baumeister et al., 2007).

Second, increasing sample size and study numbers take time and resources that may disadvantage researchers who are not at “R1” research-intensive universities. Those at smaller teaching and non-R1 institutions tend to have a higher teaching load and do not always have graduate students or postdocs to help them with research. Subject pools also tend to be smaller at undergraduate-only institutions. As a professor at a small teaching college, I therefore have limited time and resources for research. The stakes are higher for me to spend time and money in a way that can yield the best result. This practice creates disproportionate demand depending on career stage and institutional resources.

Third, I fear diversity science is continually centering majority group members’ (e.g., White, cis, straight, male, or US American) perceptions and experiences because these participants are easier to recruit. Even with online platforms, it is still difficult to recruit minorities; people may even lie about their identities (Chandler & Paolacci, 2017). These pressures can lead to a greater emphasis on the majority group members’ perspectives.

Nonetheless, this shift also led to creative solutions, often initiated by early career researchers and those at non-R1 institutions. For example, new opportunities arose that enabled unacquainted researchers to collaborate to produce high-quality publications. In particular, the Psychological Science Accelerator (PSA; Moshontz et al., 2018) enables researchers from around the world to participate in an impactful project by contributing a minimal number of participants. Because of the sheer size and breadth of the PSA network, people from small colleges or under-resourced universities worldwide could all earn authorship in top publications like Nature Human Behavior (Jones et al., 2021).

In social neuroscience, enhanced data availability has had small effects on transparency (from JB).

Long before concerns about transparency and replicability in fields like social psychology, researchers who used neuroimaging had already shifted toward making data publicly accessible. In fact, prominent journals such as the Journal of Cognitive Neuroscience went through a period where they required researchers to put their data in a public repository (i.e., 2000–2006: Van Horn et al., 2004). And a common format (e.g., Brain Imaging Data Structure [BIDS]) and many data repositories are now available and specifically aimed at cataloguing neuroimaging data sets (e.g., OpenNeuro, 1000 Functional Connectomes Project).

However, to my eye, the enhanced accessibility to data sharing in neuroscience has not, by itself, helped transparency much for a number of reasons. First, there are a wide range of acceptable approaches to conducting the many pre-processing steps which occur before GLM or other analyses. For example, one project challenged 70 research teams to analyze the same data set and essentially found that there were as many different analytical approaches as there were teams (Botvinik-Nezer et al., 2020). Even using very similar pre-processing and analytical procedures but different software can lead to slightly different results (Bowring et al., 2019, 2021). Therefore, I would argue that while neuroimaging researchers were perhaps doing a better job of sharing data, those efforts alone were not enough to strongly combat issues of transparency or bolster conclusions about replicability.

The call for data sharing has been a clear success in relationship science (from PE).

Relationship science is a subfield that has benefitted from the emphasis on making the data from published papers broadly accessible. After overcoming a few initial challenges, relationship scientists have adjusted to these new norms and launched initiatives that put these datasets to good use.

Ten years ago, it became clear that the existing APA data sharing guidelines were insufficient. The rules, of course, were clear: “After research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals…” (American Psychological Association, 2010, 8.14). Yet several studies revealed that only 20–40% of scholars provided their raw data to other competent professionals when asked (Vanpaemel et al., 2015; Wicherts et al., 2006). The overall unavailability of raw data meant that errors remained uncorrected, meta-analyses remained incomplete, and fraud remained undetected (Simonsohn, 2013).

The open science movement highlighted this problem, and it offered guidance in the form of both bottom-up and top-down solutions. On the bottom-up side, the open science movement cultivated norms of data sharing and transparency: Advocates stressed how scientists are more likely to believe conclusions if they can have access to the raw data themselves to check assumptions or run additional analyses. On the top-down side, many journals began to ask authors to make their data available during the publication process (or to explain why they could not). As a consequence of this combination of forces, datasets in psychological science were becoming more widely available by the middle of the last decade (Kidwell et al., 2016).

Like many of the open science reforms in psychology, data sharing guidelines were originally developed with modest research projects in mind: a paradigmatic example might consist of a few hundred participants and a handful of variables arranged in a single spreadsheet. But a relationship-science dataset could be quite large, involving hundreds of participants completing thousands of questionnaire items over months or years. It might involve many spreadsheets that would need to be linked together in complex ways, and it might also involve questions about sensitive topics (e.g., violence, infidelity, or intentions to divorce).

As guidelines at some major journals (e.g., Personality and Social Psychology Bulletin, Psychological Science) were beginning to change around 2014, relationship scientists needed to figure out exactly what these reforms meant for the datasets they tended to collect. When scholars elect to—or are required to—share such a dataset, what exactly do they share? Must they share all the data from the project, or something reduced? If the dataset has been reduced in some way, what can be cut without sacrificing the transparency goal? Many intelligent, well-intentioned scholars were puzzled by these questions, especially those who were not active on social media or who were not paying close attention to “the crisis.”

These issues prompted my contributions to an article that discussed how transparency solutions could and should be tailored to specific research areas (Finkel et al., 2015). This article channeled two persistent questions about the consequences of sharing the kinds of large datasets collected by relationships researchers. The first was whether the sharing of large data files might mean that researchers could be “scooped” by their own data, thereby disincentivizing future labor-intensive data collection efforts. This concern has proven to be misplaced, mitigated by the clear consensus that open data sharing policies cover the columns of data that can reproduce the published article. In relationship science, the full dataset could generate many independent publications, but this reduced dataset would rarely (if ever) generate a second publication on its own.

The second concern was that public data sharing would reveal too much about participants—to the world, and (especially) to their romantic partners. That is, a publicly available dataset might allow a motivated participant to find their own “row” and then, with a little bit of sleuthing, find data provided by their romantic partner. These issues took time to resolve, but there are indeed solutions (Joel et al., 2018). For example, data can often be hidden from participants themselves by posting aggregate scores (i.e., no item level or demographic data are posted), and data can be stored in a secure repository that requires institutional access (e.g., UK Datashare) and/or an application process (e.g., ICPSR). In some cases, id variables can be stripped out of the dataset without compromising the analysis, which makes finding one’s partner nearly impossible. In summary, it may take some extra work for relationships researchers to post their data, but these innovations mean that relationship scientists can and should participate in the new data sharing norms.

Critically, there have also been opportunities for close relationships researchers as they have made the transition to greater data sharing. Probably the most important development stemmed from the realization that relationships researchers possess a great deal of unanalyzed data. In practice, a typical close relationships dataset tends to generate only a small handful of papers before languishing as the investigator embarks on other projects. Furthermore, there has historically been no way to ascertain exactly how the measures from different datasets overlap, nor has there been an infrastructure to connect scholars with similar datasets who might benefit from combining their data.

Dr. Sara Algoe addressed these issues by establishing the Love Consortium in 2019. One of the primary goals of the Love Consortium was to inspire greater data sharing among close relationships researchers. After verifying that many close-relationships researchers had datasets that they were interested in making broadly available, Dr. Algoe created a repository through the UNC Dataverse. Nearly 100 datasets have been catalogued there, and the Love Consortium has awarded grants which foster collaborations on these datasets. These grants have already inspired many multi-study investigations that would never have happened otherwise. In short, the Love Consortium is a prime example of how norms toward greater data sharing and openness create scientific progress by dismantling inter-laboratory barriers and fostering a culture of collaboration.

How Has Preregistration Been Successful and Challenging?

The focus on pre-registering design, data acquisition, and analyses through pre-registration and Registered Reports has benefited subfields which were not originally the source of concerns about transparency.

(From JB). It has been exciting to see the effect of the open science movement on neighboring subfields such as social and cognitive neuroscience. My perspective is that the recommendation to pre-register analyses has been most beneficial to subfields involving neuroimaging because earlier efforts in those fields to facilitate data sharing were not especially successful to increase transparency or help with efforts to understand replicability.

The additional adoption of best practices like pre-registration and Registered Reports in fields like social and cognitive neuroscience has been a successful way in which Open Science has benefitted fields beyond those it was originally focused on improving (e.g., social psychology). In addition to access to data, it is equally important to understand which decisions, hypotheses, and analyses were planned in advance particularly in the case of neuroimaging analyses which includes a number of decisions that occur before analyses are even conducted. Additionally, the specificity of hypotheses can vary greatly. Currently, there are excellent templates available for researchers who want to pre-register their experimental design, pre-processing decisions, and analyses planned for hypothesis-testing when using techniques such as fMRI (e.g., Beyer at al., 2021). Many neuroscience-focused journals now accept Registered Reports (e.g., Cortex, Development Cognitive Neuroscience, Frontiers in Neuroscience, Nature Communications) whereby reviewers consider manuscripts on the basis of the strength of their methods and a priori analytical plan rather than the significance of the results.

The effort to replicate findings from articles randomly selected from prominent psychology journals (Open Science Research Consortium, 2015) illustrates why it is helpful to know precisely what authors were hypothesizing in terms of brain activation. One selected study had found that frontal lobe activation was associated with accepting unfair offers in a financial game (Tabibnia et al., 2008). The replication study was higher powered and also found that frontal lobe activation was associated with accepting unfair offers in that financial game; however, the precise location in the frontal lobes was different than the location found in the earlier study. There was a lot of discussion about whether the replication attempt should be considered successful and it essentially came down to whether we were considering the achievement of replication to be the confirmation of the hypothesis stated in the paper versus the precise coordinates of the activation in the results (i.e., the hypothesis was about a broad range of the frontal lobes rather than restricted to the specific coordinates that showed significant activation). The replication team favored interpreting the success of the replication as a function of the stated hypothesis. The hypothesis was admittedly about a broad area of the brain but they felt their job as the steward of the replication effort was to follow the published information to the letter rather than “reading the authors’ minds” as to what they may have meant. Others felt that the replication was only successful if the activation in the second study overlapped with activation found in the original study regardless of whether it overlapped with the stated hypothesized region outside of the site that showed significant activation. If pre-registration had been a best practice at the time of the original study, it may have assuaged concerns about whether the broad hypothesis stated in the article was indeed an a priori hypothesis and that specific coordinates or even specific brain hemispheres were not the intention.

Nevertheless, there are still challenges to be overcome. Support for pre-registration and replication often fails “behind the scenes.” All too often, researchers pre-register their plans and then reviewers either do not read the pre-registration or evaluate and comment on the manuscript as though it does not exist. For example, some researchers are still being told that their manuscript is not worthy of publication on the basis of results that do not show statistical significance despite pre-registration of experimental design and analyses that reviewers agree are quite strong. Or reviewers may ask whether researchers “peeked” at their data despite timestamps showing the data collection proceeded as stated in the pre-registration. Similarly, the adoption of Registered Reports as a general submission format has been very slow and often limited to very specific cases (i.e., replications of findings that must be justified as fundamentally important to the field) or relegated to journals that are not highly valued for researchers seeking jobs or tenure. The failure to adopt the Registered Report format for all higher-tier journals and for a wider range of research questions—and the rejection of manuscripts on the basis of non-significant results in the context of pre-registered robust methods—are not aligned with the promise of improving the transparency and replicability of science. There needs to be accountability in those spaces for following through on the adoption of creating more transparency through pre-registration and Registered Reports.

What Changes Would We Like to See to Help Fulfill the Potential of the Open Science Movement to Improve Psychological Science?

The three of us saw various promises and pitfalls from the past Open Science Movement in the early 2010s, but not all promises have been fulfilled and not all pitfalls have been addressed. Here, we each offer two suggestions on how we can continue to fulfill the potential of the Open Science Movement.

1. We would like to see: Continual consideration of what and who counts in open science (From JG). I am tentatively optimistic about the future of a more open and diverse science in psychology. There is active reconsideration of some practices to be more inclusive (Ledgerwood et al., 2022; Syed & Kathawalla, 2022). Building on this momentum, I hope open science continues to be reflective and become more inclusive in its meaning and participants. Open science practices have made many aspects of research accessible to those who would not normally be able to do science. But we should always be cognizant that academic cultures can preserve the status quo. Although we may consider our movements, departments, and societies to be diverse and inclusive, we risk mythologizing racial progress (Richeson, 2020). The reality is that marginalized researchers continue to be marginalized in academia (Matias et al., 2021). Creating a truly open and inclusive science requires us to actively consider our roles and positions in perpetuating or disrupting the status quo. But we cannot get there unless we reckon with the lack of diversity and inclusivity within our movements, practices, and long-held beliefs.

2. We would like to see: Changes in the incentive structure which is currently designed to undermine scientific progress and transparency (from JB). Although this may be a bit pessimistic, I don’t see the field truly getting all of the benefits of the best practices suggested to increase transparency and replicability unless the field starts centering the incentives around these practices. Right now, promotion and awards tend to be based on the contribution of an individual whereas the best science is most likely to come from teams with various expertise and, as mentioned earlier, potentially more access to collect the types of large samples or complex data which are strong building blocks of scientific knowledge (e.g., Forscher et al., 2020). In promotion and award committees, we should be asking how researchers facilitated collaborative science, inclusivity, and used transparency measures instead of simply focusing on five peer-reviewed publications to illustrate their individual contribution to science. When transparency and inclusivity considerations are incorporated at all, they tend to follow a long string of questions emphasizing the individual and are often discussed as an afterthought. Similarly, award speeches often never mention the person’s participation in transparency best practices or inclusivity efforts. Or a separate award is given as though somehow the awardee’s scientific contribution is another matter altogether.

3. We would like to see: Improvements in the motivation and opportunity to publish null or inconsistent findings (From JG). Researchers should be more willing to write up their null or inconsistent findings. Likewise, reviewers and editors should be more willing to accept such research in our journals. Null or inconsistent findings are ambiguous and difficult to interpret. However, with proper contextualization and calibration, such findings are important to our disciplines. Beyond p-values, inclusion and interpretation of other statistical information such as effect sizes and confidence intervals can offer us a richer understanding of null results.

Writing about null results requires greater sensitivity within diversity science. For instance, if we do not find majority group members demonstrating biases against a minority group, does this mean our methods were flawed (meaning it likely would not be publishable) or that times have changed and prejudice is over (unlikely given the prevalence of systemic inequality)? What happens when an expensive and time-consuming intervention field study did not yield significance? Researchers have also increasingly relied on “mini” meta-analysis to uncover a potential significant effect across a series of null findings (Goh et al., 2016). Although this meta-analytic approach offers more statistical power, I feel uncertain about the blind pursuit of p < .05. As Rosnow and Rosenthal (1989) said, “surely God loves the .06 nearly as much as the .05.”

Publishing null or inconsistent findings within diversity science is important to understand the full scope of the field, and this can only be achieved when journals are willing to publish less-than-perfect findings. For instance, Lai et al. (2016) found that most interventions could not reduce implicit biases after 48 hours, demonstrating the short lifespan of interventions and tenacity of implicit biases. Duker et al. (2022) found two common emotion regulation strategies to be ineffective in coping with gender discrimination, allowing them to consider other potential strategies that are more effective in subsequent studies. Goh et al. (2019) found that straight people did not behave differently whether their sexual minority interaction partners concealed or did not conceal their sexual orientations; the findings were carefully contextualized, Bayesian analyses were used to support the null, and the sexual minority participants’ perspectives were centered rather than over-claiming that there are no longer biases. These null results allow us to understand the full spectrum of human behaviors.

4. We would like to see: A new mindset for the pre-registration of complex data sets such as those used in relationship science (From PE). For researchers who use large and complex data sets such as those in relationship science, much more guidance about pre-registration is needed. Templates can be helpful, of course (e.g., van den Akker et al., 2021), but I would argue that the real challenge is conceptual: We need to disentangle the various practices that were conflated in basic experimental preregistration.

To start a broader conversation, I will describe what I believe to be helpful for close relationships researchers who are interested in preregistration. Critically, I start by assuming that we can separate out the various amalgamated “preregistration” practices because these practices serve different goals (da Silva Frost & Ledgerwood, 2020; Ledgerwood, 2018, 2019). In most cases, I have found that it is possible to separately consider an easy goal (transparency), an intermediate goal (control Type-I error), and a hard goal (test a theoretical prediction). I tackle the goals in sequence (i.e., only attempt the hard goal if you were also attempting the easy and intermediate goal).

The easy goal is “be transparent about my data analytic decisions.” This goal is achieved by describing various data processing decisions that will likely come up, and by explaining what statistical tests you will run. As best you can, create the table that will ultimately go in the manuscript, and leave placeholders (e.g., x.xx) for the values; this task will likely prompt you to think about all the different ways you could operationalize each variable before you actually start conducting analyses. One can easily do this kind of preregistration in stages: If during data processing you encounter a decision you hadn’t anticipated, decide what to do, then update the preregistration before proceeding further (an example can be found in this article: Eastwick et al., in press). In my experience, this easy goal is almost always achievable.

The intermediate goal is “control my Type-I error.” This is more challenging than a simple two-condition study. It is challenging in part because you may be carrying out many tests of a hypothesis (e.g., you are examining the effects of 3 correlated independent variables on 2 correlated dependent variables); in such cases, it will be important to specify how you will control family-wise Type-I error rate. I tend to favor a Holm-Bonferroni test (Holm, 1979) when it would be conceptually meaningful that some tests might receive support and others would not (e.g., these five traits had significant effects, and these five traits did not). I tend to favor a binomial test (Bahns et al., 2017) when it is conceptually meaningful to have an “up or down” decision about whether the hypothesis was supported across all tests (e.g., across all operationalizations of a given statistical interaction, the number of significant effects did not exceed what would be expected due to chance). A related challenge is that you might have existing knowledge about a dataset. Here, I suggest transparently reporting what you know about the dataset already; it is not clear whether there is a formal way of controlling Type-I error in this circumstance, but at least readers would be able to evaluate for themselves if your effects are overfitted.

The hard goal is “test an a priori prediction derived from theory.” For many scholars, this is exactly what preregistration means: recording a prediction ahead of time (Nosek et al., 2018). In direct contradiction to this convention, I tell close relationships researchers to avoid the word “predict” entirely when addressing the easy and intermediate goals above; only use that word to address the hard goal (and there is nothing wrong about addressing only the easy and intermediate goals). In the large intensive datasets used by close relationships researchers, prediction can sometimes boil down to a person’s own intuitive hunch. There is nothing wrong with recording these hunches ahead of time—you will learn about how good your hunches are!—but your a priori hunches do not affect Type-I error, nor do they test a theory (da Silva Frost & Ledgerwood, 2020; Ledgerwood, 2018, 2019). If you want to test an a priori prediction, describe how the hypothesis derives from a theory, and describe how you (or another reasonable scholar) would adjust your confidence in the theory upwards or downwards depending on the results. I usually only advocate the pursuit of the hard goal if the study design permits you to adjust your confidence up or down; long-shot, counterintuitive studies that are interesting only if they “work” are usually bad candidates for the hard goal of testing an a priori prediction derived from theory.

We need to pull apart the myriad goals that preregistration can achieve and devise better guidelines for scholars who work with datasets which do not easily conform to the basic experimental preregistration schema. I find that the tripartite structure described above gets 80% of the way there.

5. We would like to see: The field getting serious about transparency through a new approach to pre-registration and a wider range of Registered Report opportunities (from JB). One significant change I would like to see with the aim of increasing transparency is a new approach to pre-registration and the opportunities to submit to peer-reviewed outlets in the form of Registered Report. I think of pre-registration as a contract that scientists have with the community. I am open to reading any analyses that authors have conducted on their data; however, I do want to know at what stage they planned to conduct those analyses and whether they already knew other results at the time. The adoption of Dr. Eastwick’s easy goal described earlier in this article is a great way to encourage pre-registration that is not too restrictive. For example, my lab regularly pre-registers analyses in addition to those focused on testing our hypotheses. In our pre-registration document, we often include analyses we know we want to conduct to explore the data but may not have a specific hypothesis we want to test. Or we may want to conduct follow up tests to more fully understand the data set but these analyses sometimes depend on initial results. Therefore, we label the analyses as exploratory or note that we plan to conduct a series of analyses if the initial results conform to a specific pattern. Then, it is clear what was (and was not) planned as in cases where scientists conduct follow-up studies based on an exploratory finding (e.g., Munin & Beer, 2022) or are asked to do additional analyses (e.g., Freedman et al., 2018).

I also expect that editors and reviewers will take pre-registration seriously rather than ignoring it or asking authors if they really did what they pre-registered unless there is some type of discrepancy in the record. There has to be some degree of trust with pre-registration—otherwise, we run the risk of undermining the use of archival data sets. We cannot argue that people should focus on data sharing on the one hand but then make it difficult for people to publish from archival data on the other hand because they cannot really “prove” that they did not conduct any analyses before posting a public pre-registration analyses plan. Editors and reviewers don’t operate in a vacuum; the others involved in the review need to engage and question why pre-registrations are not honored or ignored to ensure that the field realizes their full value. Without the flexibility to fully delineate a plan or the honoring of the pre-registration in the absence of discrepant information, pre-registrations begin to feel like traps or worthless.

Additionally, I would like to see Registered Reports become the default manuscript submission format at peer-reviewed outlets. It does not need to be the only acceptable format but I would like to see it centered as a primary and widely available format. The restriction to certain types of studies such as only replications and the need to justify a higher bar of novelty or importance than other types of submissions does not align with the improvement of science. If the goal is to evaluate science on the basis of the research question, methods, and analyses rather than whether the results were significant, then we need the primary form of evaluation to match those elements of the research. Additionally, Registered Reports promise to facilitate the dissemination of results rather than hamper them. As many trainees are now seeking careers outside of academia, their later stage research often fails to be published as interest in academic publication can wane quite considerably. However, if trainees were only collecting data with a pending agreement for publication of those results, then results (significant or not) of data collection would be more likely to end up being shared as trainees near graduation would be tasked with adding results and discussion to an accepted manuscript rather than starting the manuscript process from scratch. On a more context-specific note, the adoption of Registered Reports as a standard format would also aid the dissemination of science when access to data collection may be restricted or limited in time. For example, in the early stages of the pandemic, many labs had to freeze data collection. This time could have been spent submitting and reviewing Registered Reports which then could be acted upon as access to data collection become more feasible in between waves of the virus.

6. We would like to see: Enhancing the potential of open data by modeling off of the Love Consortium (from PE). If the goal of making datasets openly available is to allow other researchers to verify original authors’ claims, then the current approach—whereby journals require authors to make a dataset available in a repository for other scholars to inspect—is probably sufficient. But bear in mind that the key innovation of the Love Consortium dataverse is that it was a promotion-focused rather than prevention-focused endeavor. That is, it was formulated to address the question “What more could we be doing with these datasets?” as opposed to “How can we double-check existing publications?” As a result, Dr. Algoe and colleagues designed the Love Consortium dataverse to facilitate collaboration and discovery by bringing different datasets together in a novel and generative way.

Scholars who work with large datasets in other areas of psychology can benefit enormously from this approach. Imagine that a researcher is interested in the correlation between two variables, and they sense from the published literature that this correlation tends to be higher in sample type A than in sample type B. If their subfield maintained a similar dataverse that described all measures administered in each study (whether or not those measures were in the published manuscript), it could be easy for the researcher to search all the datasets that permitted the calculation of this correlation. If enough other datasets contained the correlation of interest, the researcher can calculate more precise effect size estimates than if they had tried to collect the data themselves. In summary, the Love Consortium dataverse illustrates how open data can be leveraged to achieve new discoveries, all the while maximizing statistical power and making efficient use of languishing datasets.

Summary

As three researchers who differ in our career stage, research environment, identities, and subfields, we see many successes of the open science movement but also note that implementation has trailed behind its full potential. We all feel encouraged by the focus on larger samples, greater data sharing, and the use of pre-registration for experimental designs as well as analytical decisions for investigating planned and archival data sets. However, there are a number of areas that will help psychological science move closer to correcting questionable research practices, bolstering replicability, and regaining credibility. Overall, there is potential for the open science movement to become a strong ally in fostering diversity, equity, and inclusion. Additionally, more careful consideration of how to facilitate data sharing and pre-registration could be useful, perhaps accompanied by a shift in the incentive structure of our field.