Open science can enhance research quality and collaboration

Interview with Dr Ineke Wessel, associate professor in Experimental Psychopathology at the Faculty of Behavioural and Social Sciences

“Whenever I make something open – be it my data or methodology – I am always a bit nervous before I click the button. It’s normal. This is due to the culture that prevails in academia whereby, as a researcher, you think you are not allowed to make mistakes.”

Dr Ineke Wessel studies autobiographical memory, including emotional memory. She is a member of the Exceptional Sex Offenses National Expertise Group (LEBZ), provides postdoctoral training to practitioners and coordinates the Research and Innovation course which is part of the PPO training institute’s specialization programme in Clinical Psychology.

Recently, she authored a series of blog posts about her experiences with open science and, as of this year, will be co-organizing ReproducibiliTea meetings at the Faculty of Behavioural and Social Sciences. 

What made you an open science convert? 
It was a process that happened over many years. Since the beginning of my career, I haven’t felt fully comfortable with certain practices. For instance, in psychology it used to be common to do studies using small sample sizes, and some researchers would be pretty lenient with the way they reported their findings. But at the beginning, I didn’t question these practices enough to do anything about them.

I began to question my research practices more when I started to read about open science developments. One of the things that really struck a chord with me was a lecture by Zoltan Dienes, based on Gelman and Loken’s famous paper that compared data analysis to a garden of forking paths. The paper illustrates the huge number of choices that researchers make when selecting and analyzing their data – and how each of these choices could potentially lead to different statistically significant outcomes. This was eye opening for me and really changed my perspective. Another read that resonated with me was The Seven Deadly Sins of Psychology by Chris Chambers.

Also, I was a direct witness of the Stapel affair. I was on the examining committee of one of his PhD students and was shocked to find out that the data discussed in the student’s thesis was fabricated. I hadn’t noticed and neither had the student. However, blatant fraud like this is very rare. In most cases, questionable research practices are unintentional. 

What do you think is needed to make science more transparent? 
I am a big proponent of pre-registrations (see glossary below). I think that researchers should be as detailed as possible regarding all the decisions they intend to make during data collection and analysis: things like what exclusion criteria they will use, when they plan to stop their data collection and what comparison they want to do to test their hypothesis. All these decisions can be made before collecting the data.

While reviewing a paper recently, I noticed that participants were excluded from the analysis based on the data. I consulted the corresponding pre-registration information and found that the rationale for the exclusion of these participants hadn’t been specified. This made me question the basis upon which these decisions had been made.

While a researcher might have the best intentions, we all have confirmation biases and it is hard to ignore them. As humans, we are naturally prone to choosing the solution that fits best with what we believe.

What are the obstacles to open science?
Currently, the system of incentives regarding career progression focuses entirely on the end product: publications. However, the number of publications or high-impact publications should not be the only thing that matters when it comes to promotion criteria. Pre-registrations or registered reports should be rewarded too. I know that these issues are being discussed in the Netherlands and luckily, things are changing. 

As far as I am concerned, I am in a comfortable position in my career because I don’t need to have first author publications so badly anymore. Early-career researchers need them to build up their CVs and advance in their careers. They are the ones that are most disadvantaged by the current system of rewards and incentives. 

Do you teach about open science? 
I teach the deepening module in Psychology as part of the Honours College. Most of the students following this module are already familiar with open science and the replication crisis since these topics are covered in other course units. In my classes, we expand on these issues by discussing how the replication crisis affects the publishing and reviewing processes. I have been an associate editor for a journal and witnessed first-hand examples of p-hacking and other questionable practices. I discuss these experiences with my students and they generally appreciate getting an inside look and hearing anecdotes about these things.

Recently, I’ve also started to teach in the specialization programme in Clinical Psychology at the PPO training institute. Here, I plan to discuss pre-registrations right at the beginning of the curriculum. I would like to have the trainees practice with pre-registrations and realize the importance of planning before doing the data collection.  

What are the advantages of being more open?
The open science movement is taking a bit of pressure and competitiveness off science. For instance, I find it relieving that it is OK to be working on the same topic as someone else and that your research doesn’t always have to be novel. I am now working on a meta-analysis and know that there are two other groups that are working on the same topic. Ten years ago, I would have been very annoyed about this but now, I think that it can be valuable because these projects can complement each other. If the studies end up not being complementary, that’s also OK because then we are unintentionally replicating each other's meta-analysis. 

On the other hand, I completely understand that people find it hard to be open about their research process. Whenever I make something open – be it my data or methodology – I am always a bit nervous before I click the button. It’s normal. This is due to the culture that prevails in academia whereby, as a researcher, you think you are not allowed to make mistakes. If you make an error in your data collection, analysis or methodology, the general tendency is to hide that something went wrong. We are humans and it’s impossible not to make mistakes. I think we should accept this and be transparent about the mistakes that we make. I believe that openness and transparency are the best ways forward to improve our practices.

Most open science enthusiasts are early-career researchers. How do you experience being a senior researcher in this community?
I wish that the distinction between junior and senior researchers was not so prominent among the open science crowd. For instance, this summer I was at the conference of the Society for the Improvement of Psychological Science in Rotterdam, and at some point there was a survey among the audience about how old people were. I was one out of five people in the room who were aged 50 or older. Most people were PhD students, postdocs or early-career researchers. Although this was a very inclusive group, I felt awkward to find myself in this position.

During the conference, I attended a workshop about meta-analysis and during the theoretical part, I was able to participate and offer some advice. After this, there was a hands-on workshop where everyone turned on their laptops and started using the open-source statistics program R. The workshop host asked who was not familiar with R and I was the only one in the room that admitted to not knowing it. This is because of the technological advantage that younger generations have. As a senior researcher, it is difficult to keep up with these advancements because you have less time due to teaching and administrative commitments. I wish that more senior colleagues would think not only about the challenges of our career stage (such as the technological disadvantages) but also about the opportunities as well. We can actively participate in the change to a more open way of working.

Glossary
Pre-registration: When you pre-register your research, you specify your research plan in advance of conducting your study and then submit this to a registry. 
Replication crisis: This refers to the fact that a large proportion of published scientific findings cannot be replicated/reproduced. 
P-hacking: Also known as data drenching or data fishing. This refers to the practice where researchers consciously or unconsciously manipulate data until their analysis yields statistically significant findings.