Winner 2023 - Organizing a Hackathon to Develop a (Pre)registration Template for Studies Using Passive Smartphone Measures
Open Research objectives/practices
Making scientific research more reproducible by increasing the amount and quality of information placed on the public record.
Creating new tools or technologies to facilitate Open Research practices.
Introduction
In our research, we have repeatedly utilized passive smartphone measures, such as data on app usage, calls, text messages, and wifi connections. These measures have gained popularity over the past years due to their ability to capture an individual's behavior with a low participant burden. However, passive smartphone measures introduce methodological challenges during data collection and analysis. For example, there are many decisions that researchers must make when working with passive smartphone measures (such as data cleaning and analysis), which are often not reported transparently. Notably, previous research has demonstrated that various decisions made during the research process can yield different conclusions (e.g., Cai et al., 2018; Niemeijer et al., 2022).
To enhance the transparency of studies utilizing passive smartphone measures, the incorporation of open science practices is an important next step. Recently, researchers started to point out the importance of preregistration as a means to improve the quality and usefulness of those studies (Davidson, 2022; Velozo et al., 2022).
Motivation
Despite the recognition of the usefulness of preregistering studies using passive smartphone measures, our endeavor to preregister our own study revealed difficulties arising from the multitude of decisions involved.
In contrast to other research fields where templates exist to assist researchers in considering specific decisions ahead of time (e.g., Kirtley et al., 2021), a comparable template is currently missing for passive sensing studies. This absence poses an additional hurdle for researchers who want to preregister their study plans. Thus, our primary goal was to develop a preregistration template and accompanying tutorial specifically for passive smartphone studies. This will make it easier to preregister upcoming projects and provides a valuable resource for fellow researchers in the field. To do so, we organized a hackathon at the Society for the Improvement of Psychological Science (SIPS) conference 2023. Currently, we are working on writing up the tutorial paper and finalizing the template.
Lessons learned
We learned that preregistration is not so common yet in studies using passive smartphone measures, and that it comes with its own challenges. For example, the data cleaning is more challenging and often more complicated analytical methods are used, such as machine learning models, which require various decisions. All such decisions are difficult to anticipate beforehand.
Organizing a Hackathon presents challenges in finding a suitable location to host it; however, the existence of conferences like SIPS played a crucial role in making it possible. Unlike many other conferences, SIPS offers the opportunity to host Hackathons or workshops, which facilitated the planning and execution process.
Additionally, it is difficult to engage experts in relevant fields to participate. Our aim was to involve researchers who had prior experience working with passive data across various disciplines to gather diverse opinions and to include all crucial decisions in our template. Thus, prior to the Hackathon we reached out to various experts that shared the invitation within their network. Additionally, the SIPS organizers allowed researchers to participate in the Hackathon without paying the conference fee, which helped to include researchers that did not plan to join the whole conference.
Fortunately, the Hackathon was successful with around 20 people joining and collecting many important considerations that should be included in the template. Currently, it is challenging to organize the writing process with such a large group. Thus, we are planning a follow up meeting to divide the writing tasks.
URLs, references and further information
References
Cai, L., Boukhechba, M., Wu, C., Chow, P. I., Teachman, B. A., Barnes, L. E., & Gerber, M. S. (2018). State affect recognition using smartphone sensing data. Proceedings of the 2018 IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies, 120–125. https://doi.org/10.1145/3278576.3284386
Davidson, B. I. (2022). The crossroads of digital phenotyping. General Hospital Psychiatry, 74, 126–132. https://doi.org/10.1016/j.genhosppsych.2020.11.009
Kirtley, O. J., Lafit, G., Achterhof, R., Hiekkaranta, A. P., & Myin-Germeys, I. (2021). Making the Black Box Transparent: A Template and Tutorial for Registration of Studies Using Experience-Sampling Methods. Advances in Methods and Practices in Psychological Science, 4(1), 2515245920924686. https://doi.org/10.1177/2515245920924686
Niemeijer, K., Mestdagh, M., & Kuppens, P. (2022). Tracking Subjective Sleep Quality and Mood With Mobile Sensing: Multiverse Study. Journal of Medical Internet Research, 24(3), e25643.
https://doi.org/10.2196/25643
Velozo, J. D. C., Habets, J., George, S. V., Niemeijer, K., Minaeva, O., Hagemann, N., Herff, C., Kuppens, P., Rintala, A., Vaessen, T., Riese, H., & Delespaul, P. (2022). Designing daily-life research combining experience sampling method with parallel data. Psychological Medicine, 1–10.
https://doi.org/10.1017/S0033291722002367
| Last modified: | 01 November 2023 11.08 a.m. |