Reanalyzing openly available data

Open Research objectives

• Using online tools and services to increase the transparency of research processes and methodologies.
• Making the outputs of research, including publications, data, software and other research materials freely accessible.

Introduction

In 2019, Quoidbach and colleagues published a paper in a highly prestigious journal of psychology. Laudably, they also made the data of their study publicly available on the website of the Open Science Framework (osf.io/bxgn4).

As the paper promised to be highly relevant for my own research (on social behavior and well-being), I read the paper with great excitement. However, I quickly became skeptical about one of the main findings (i.e., that happiness positively predicts solitude a few hours later). Because the data was openly available, I was able to consult the raw data. The data indicated that on a descriptive level, the prediction was the opposite (i.e., that happiness negatively predicts solitude a few hours later).

After some more investigation, I found out that a modelling artifact was responsible for the inconclusive results, which the original authors did not report. I reported all of this in a commentary paper that was later published in the same journal as the original study. I also made a preprint of the paper and the analysis scripts publicly available (osf.io/zk98q), thus making all the materials of the reanalysis freely accessible. I used the publicly available dataset and online tools of osf.io as means to increase the transparency of research findings. Specifically, I took a close look at what was happening “under the hood” of a well-received empirical finding. By showing that the claimed association is inverse under other model specifications and in the raw data, I contributed to transparent practices within psychological science. My commentary paper further evoked a theoretical and methodological process in myself and the original authors. Some of the results of this process can be found in the reply paper by the original authors (Quoidbach et al., 2020).

Motivation

It was important for me to acknowledge that the original authors of the paper made the data publicly available. So they set the stage for a transparent reinvestigation.

In the revision process of my commentary, it was very important to me to further contribute to this transparency and make my analysis script available to the reviewers (two anonymous ones and the original authors). Upon the publishing of the paper, I also made the analysis scripts publicly available so that other scholars could evaluate and reproduce my statistical work.

I think this commentary (and the reply paper of the original authors) is a nice example of how open data practices can lead to a scientific exchange that fosters progress on a theoretical and methodological level. The journal editor later praised this example on Twitter as such an example.

Lessons learned

I learned that reanalyzing other people’s data can be a lot of fun and contribute to an in-depth understanding of methodological and theoretical details of a paper. I have to admit that I was quite scared to confront the original authors with my reanalysis. I was glad to have (other) statisticians around that would confirm the correctness of my analysis. To my pleasant surprise, the original authors were very friendly and appreciative of my reanalysis.

Since then, I sometimes download the openly available data of papers that I read. I then “peak under the” hood of their analysis and look at the raw data and their statistical models. It always helps to understand the paper and the data better.

URLs, references and further information

• https://osf.io/zk98q
• https://osf.io/bxgn4
• Quoidbach, J., Taquet, M., Desseilles, M., de Montjoye, Y.-A., & Gross, J. J. (2019). Happiness and Social Behavior. Psychological Science.
• Quoidbach, J., Dercon, Q., Taquet, M., Desseilles, M., de Montjoye, Y. A., & Gross, J. J. (2021). Happiness and the Propensity to Interact With Other People: Reply to Elmer (2021).