Each month, the open access team of the University of Groningen Library (UB) puts a recent open access article by UG authors in the spotlight. This publication is highlighted via social media and the library’s newsletter and website.

The article in the spotlight for the month of April 2023 is titled The process of replication target selection in psychology: what to consider?, written by a research team led by Merle-Marie Pittelkow (PhD student at the department of Psychometrics & Statistical Techniques at the Faculty of Behavioural and Social Sciences).

Abstract

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

We asked corresponding author Merle-Marie Pittelkow a few questions about the article:

This article is a registered report, which means that methods and proposed analyses were pre-registered and peer-reviewed prior to the research being conducted. Why did you choose a registered report?

Personally, I am a big fan of the registered report format. I liked that it forced us to specify our aims and methods in detail before we conducted the study. I also think that peer review prior to data collection is much more useful than "after the fact", as you still have the opportunity to change aspects of the methodology rather than having to argue later on why a suboptimal design might still lead to valid conclusions. And lastly, the registered report format, to me, is the most transparent way of conducting a study and communicating the research process. 

In this project specifically, I was also attracted by the challenge of registering a mixed methods study (qualitative and quantitative data collection and analysis) with an exploratory aim. I wanted to demonstrate the applicability of the registered report format for this type of research. To my knowledge, this had never been done before (at least not at RSOS, as the editor confirmed) and I wanted to set an example for how registered reports could also be useful for studies that are not confirmatory.

You involved the scientific community in your research, and argue that this involvement was crucial. Can you explain why? 

I believe that any effort to propose reform or create tools benefits from the involvement of its target audience. In order to create something that is applicable and useful, it is crucial to first understand the target behavior you are trying to streamline or change. For example, we [the author team] had made suggestions for how to select replication targets in the past (Field et al., 2019; Isager, 2019; Isager et al., 2021; Pittelkow et al., 2021). These suggestions were guided by our personal experiences of selecting replication targets and/or assumptions about how replication studies ought to be selected. As a consequence, the suggestions we had put forward were heavily shaped by our own perspectives, and not necessarily applicable to a more general target population. In this project, we aimed to create a tool that would be useful to a broader target population (psychological researcher) and wanted it to (in part) reflect how replication targets are selected in practice while validating each item through expert discussion. To this end, we involved the target community by (1) conducting a registered report, (2) surveying researcher’s current practice of replication target selection, and (3) asking experts to discuss and validate the suggestions put forward.

The replication crisis has increased awareness about the importance of replication studies, but the number of these studies is still very small. In your article you mention an estimated replication rate in the social sciences and psychology of just 1%, which means that for 99% of the research no replication studies are being done. What can be done to improve this?

I think change can only be achieved with the help of various stakeholders within the complex scientific system. In the end it boils down to redefining the value of research and changing the incentive structure within academia. There have been promising changes in the past including the widespread adoption of registered replication reports at journals, funding schemes targeting replications, or the integration of replication studies in University education (e.g., as thesis projects). Nonetheless, the novelty of a study/finding retains its high value with publishers, funders, or hiring committees, which consequently means that researchers will prefer new studies to replications. Moreover, the focus that is put on ‘first-author’ contributions in evaluating the performance of individual researchers, further disincentivizes participation in large-scale replication projects, as the important contribution of individual researchers in these large team projects is often overlooked by relevant stakeholders (i.e., hiring committees). But here too, we do see changes in the academic system with many stakeholders now advocating for more ‘team science’. I am curious to see if this will bring a shift to how we currently value and conduct research.

I think that within the next couple of years, we will also have to be mindful of the fact that replications might not play the same role across all methodologies and disciplines. As such there might be unintended consequences for some, if replications become too valued. But this might be more of a cautionary afterthought.

Could you reflect on your experiences with open access and open science in general?

Throughout my PhD, I have made an effort to conduct my research projects as transparently as possible. Thiswhich included, creating preregistrations/registered reports, preprints, and publishing my papers open access. Personally, I believe that I have benefitted from these open science practices. Preregistration made me think about the aim of my research projects and how my methods would map onto them more deeply, preprints made my research available timelier and had me connect with other researchers without the 6+ month backlog that the traditional publishing system would bring. As a final consequence of publishing open access, my research is accessible to the wider community of research consumers.

Useful links:

Merle-Marie Pittelkow’s personal website, that provides information about her research projects, publications and more.

Open Science Framework: a platform maintained by the Center for Open Science that helps researchers conduct research more rigorously, and manage and share their work more openly.

Citation:

Pittelkow, M-M., Field, S. M., Isager, P. M., Veer, A. E. V., Anderson, T., Cole, S. N., Dominik, T., GinerSorolla, R., Gok, S., Heyman, T., Jekel, M., Luke, T. J., Mitchell, D. B., Peels, R., Pendrous, R., Sarrazin, S., Schauer, J. M., Specker, E., Tran, U. S., ... Ravenzwaaij, D. van. (2023). The process of replication target selection in psychology: what to consider? Royal Society Open Science, 10(2), [210586]. https://doi.org/10.1098/rsos.210586  

If you would like us to highlight your open access publication here, please get in touch with us.