• This case study is one of the three winning submissions for the Open Research Award, which will be celebrated on 22 October 2020.

The ESM Item Repository is introduced as a platform to improve transparency, methods reproducibility, and validity within ESM research. Experience sampling methods (ESM) have become increasingly popular to investigate psychological phenomena in ecologically valid, daily life settings. ESM allows researchers to measure psychological constructs such as mood multiple times a day for long, continuous periods of time. However, due to this popularity researchers often had to develop their own ESM items to assess specific constructs, as initially there were not many validated ESM item batteries available. This has caused somewhat of a wild growth of ESM items. Moreover, exact ESM item wordings and scales are often not provided in publications, which has caused a loss of overview of what items are commonly used and how valid such items are. Hence, there are a number of broader issues that currently hamper ESM research, for example; a lack of transparency in what items are used, difficulties replicating ESM studies due to a study measures not being available, wasting research resources due to having to develop ESM items which might already exist, and a lack of insight into item validity.

Recognizing these challenges for ESM use in psychological sciences, during the Open Science hackathon at the 2018 BelgianDutch ESM Network, our team* identified that making an overview of ESM items openly available to all researchers was a cornerstone to improving transparency, methods reproducibility and validity within ESM research. Shortly thereafter we started contacting ESM researchers to collect ESM items they used. Due to massive interest and support from fellow ESM researchers, we quickly gathered hundreds of ESM items, thus necessitating the creation of an openly accessible online repository. Therefore, on 8 October 2019 the esmitemrepository.com domain was registered to serve as a home for the new item repository. However, to make the repository work and accessible for researchers worldwide we had to solve a number of challenges ourselves. For example, collecting such large numbers of ESM items in a standardised manner from various research groups required the development of a standardised item input format. Moreover, harmonizing and organizing these ESM items in a way which allows them to be presented to other researchers proved quite challenging.

Furthermore, providing an intuitive graphical user-interface which allows for straightforward access to the complete dataset, and creating awareness about the repository beyond our own immediate network are the focus of continuous improvement. By leveraging the capabilities of the R Shiny package for creating appealing user-interfaces and interactive web applications, and the Open Science Framework (OSF) for providing transparent and open data hosting, we were able overcome a number of challenges and kick-start our new repository. Now, only 10 months later, we were able to collect and make publicly available more than 750 ESM items, serving over 4400 visitors since its inception. Moreover, we are still expanding and improving the repository; ESM items are still being collected and plans for ESM quality assessment strategies are being created.

As such, the ESM item repository is a unique Open Research project that publicly presents research output – the ESM items – by applying Open Science principles in practice. By doing so, the repository helps researchers to find ESM items, learn about their usage, and helps identify relevant items for the construct of interest or overlapping items. It also allows researchers to increase the openness of their own ESM research, by making their ESM materials findable via the repository. The infrastructure created by the repository furthermore supports conceptual development and psychometric evaluation of included ESM items. As such, it showcases how applying Open Science practices can help improve existing scientific processes.

Team composition:

Olivia Kirtley (KU Leuven), Anu Hiekkaranta (KU Leuven), Yoram K. Kunkels (University Medical Center Groningen), Gudrun Vera Eisele (KU Leuven), Martine van Nierop (KU Leuven), Davinia Verhoeven (KU Leuven), , and Inez Myin-Germeys (KU Leuven).

References and further information:

• ESM Item Repository
• The Experience Sampling Method (ESM) Item Repository