Introduction, description of the research context, and what open practices were used and why:

In this study the goal is to see if Early Warning Signals can be used to predict upcoming transitions in depression. This study required a large amount of measurements over a period of 6 months (~592 for each participant). Due to the difficulty in collecting such data, only a total of 7 participants had been studied before the current study (1). In previous studies, analytic choices had to be based on theory alone, and no golden standards existed. For the current study, analytic choices were made by using data obtained in these previous studies as pilot data, and simulating data based on these previous datasets. Making these choices and the reasoning behind these choices publicly available in a preregistration [2], is especially valuable due to the very limited amount of previous studies with similar designs. By performing many alternative analyses at each point where an analytic choice was made (not unlike a multiverse design), and preregistering what our a priori hypotheses for the results of these alternative analyses is, we share as much information as we can to work towards a golden standard for analyses in this field.

Challenges encountered, how these were handled, and benefits realized:

Performing many alternative analyses using the same data raised some challenges:

1. Based on previous literature, pilot data, and simulation studies we performed, some analytic choices were a priori known to be more sensible than others. The results from these sensible analyses may be obscured if all alternative analyses would be treated as though they had equal prior weight.
2. A subset of the alternative analyses is expected to randomly work well based on chance, this subset should not be overinterpreted post hoc.
3. Fully understanding implications from an extensive range of results may be difficult for a broad audience of readers.

Though it is hard to put a sensible prior weight on all analyses in a multiverse, in this case study it was possible to propose a ‘main analysis’ to achieve a similar goal. In this main analysis, all analytic choices were made in such a way that based on previous literature, pilot data, and simulation studies, we would have the highest a priory chance of obtaining sensible and accurate results. By specifying this ‘main analysis’ it becomes clear that alternative analyses more similar to this analysis should receive higher prior weight when interpreting the results, than alternative analyses that are very dissimilar from this analysis.

Alternative analyses that would be obtained if these choices were made differently, will be executed according to a preregistered analysis plan that has been published online (2). In addition, we preregistered why we expected each alternative analysis to be inferior to the main analysis, and made predictions on how and why the results would change and weaken when an alternative analysis approach is taken. This allows sharing the results of alternative analysis without the need for post hoc interpretations of alternative results.

Finally, we believe this ‘main versus alternatives analysis’ structure makes the results easier to interpret for readers because it provides a baseline, while still providing as much information we can to improve future analyses using similar data.

Lessons learnt, and conclusion:

Though this kind of extensive preregistration took a lot of time, it was also be a nice outlet to share the knowledge that went into making all these analytic decisions in the first place. The ‘main versus alternatives analysis’ structure could be helpful to reduce some challenges encountered in multiverse studies.

References

1. SMIT AC, SNIPPE E, WICHERS M. Increasing Restlessness Signals Impending Increase in Depressive Symptoms More than 2 Months before It Happens in Individual Patients. Psychotherapy and Psychosomatics. 2019:1-3.
2. SMIT AC, SNIPPE E, KUNKELS YK, RIESE H, HELMICH MA, WICHERS M. Transitions In Depression (TRANS-ID) Tapering. Retrieved from osfio/h75p9. 2020.