From development to sustainable open-source software: an interview with Sebastiaan Mathot on OpenSesame

Sebastiaan Mathot is an assistant professor at the University of Groningen's Department of Experimental Psychology. In this interview, we delve into his journey in developing OpenSesame – an open-source experiment builder – to understand the challenges and successes he encountered, and gain insights into the broader implications of his work in the context of research software management and sustainability.

OpenSesame has gained significant attention and widespread adoption, with more than 50.000 downloads and a publication in Springer with 34.000 accesses and more than 2.500 citations. Can you describe to people unfamiliar with this kind of software what OpenSesame does?

OpenSesame is a tool for building behavioral experiments of the kind that are often done in psychology, cognitive neuroscience, and experimental economics. Such experiments resemble simple computer games in which participants, which can be humans or other animals, see or hear something, and then they respond to this by pressing a button or performing some other action. Eye movements, brain activity, or other relevant measures are sometimes also recorded. OpenSesame allows you to build such experiments easily by combining a graphical user interface with scripting in Python or JavaScript.

What inspired you to create OpenSesame?

As part of my PhD I did a lot of these kinds of behavioral experiments, often combined with eye tracking. In our lab at the VU University Amsterdam, the default software to build and run these experiments was a program called E-Prime. And sometimes we also implemented experiments directly in C++, a fairly low-level (and thus complex) programming language.

I always enjoyed programming, and it seemed like a fun exercise to try and build my own software to build and run experiments. As far as inspiration goes, that was essentially it. OpenSesame was not part of a larger project. I was not hired to do it. And I certainly did not expect it to become as big as it did.

Why did you choose to make OpenSesame an open-source project? What benefits do you see in the open-source model for scientific software?

I have mostly been using open-source software myself for a long time. So making OpenSesame open-source as well seemed like the natural thing to do.

A key benefit of open-source is that resources go to activities that add value. This sounds abstract, but it can mean a few things. For example, companies that sell closed-source software often spend most of their budget on advertising, reaching out to customers, patents, etc. These activities do not add value in the sense that they only serve the interests of the company, but not of society more broadly. As a customer who is buying a closed-source product, you’re not only paying for the development of the product, but also for these other activities, even though they don’t benefit you.

Closed-source software also tends to duplicate effort, which again doesn’t add value. For example, a closed-source program might not be allowed to make use of certain open-source libraries that are needed, because the open-source license might prohibit it. (Whether it does or not depends on the specifics of the situation, but it can happen.) This means that the developers have to re-implement this functionality themselves. And as a customer, you’re paying for this duplication of effort.

Open-source software also makes it transparent how the software works. In many cases, this is not of primary interest to users of the software. But in some cases it can be, especially when software deals with sensitive data or is used in key infrastructure.

Finally, open-source software makes it easier for other people to contribute to the development of the software. This is often touted as the raison d’être of open-source. But in practice it doesn’t happen very often, as I will describe later, so the practical importance of this point is debatable.

Under the right conditions, open-source software has a lot of benefits. However, the right conditions are often not created by universities and research institutions. This is to the detriment of everyone except commercial companies, who of course are happy to step in to take advantage of this.

How do you manage contributions from the community, and what role does community feedback play in the development of OpenSesame?

OpenSesame is open-source, which means that in principle anyone can review the code, find bugs, and submit improvements. But in practice, this is really difficult, because the code base is large, complex, and spread out over many different repositories. Contributions also have to be properly tested and make sense in the larger development plan. As a result, community contributions to the code itself are rare. This reality is a big departure from the idealized picture that many people have of open-source software.

In the end, core development is handled by two or three developers who work on a different aspect of the source code. I work on the core of OpenSesame itself. Kristian Lange works on JATOS, which is a server for managing online experiments; this is a separate project, but we work closely together because the two projects feed into each other. And in the past, Daniel Schreij and Jaap Bos worked on OSWeb, which is a JavaScript implementation of OpenSesame.

As mentioned, ad-hoc community contributions don’t play a big role in core development, but they do play a role in other areas. Some users develop their own plugins. These have an independent code base, which means that it’s relatively easy for users to work on them. For example, university IT departments often develop plugins for devices that are specific for their labs. The UG has done that too. Users also contribute tutorials or other forms of documentation. And they help each other on the support forum.

Developing open-source software like OpenSesame involves significant challenges, especially in terms of maintenance. How do you ensure the long-term reliability and maintenance of OpenSesame? 

I have yet to find a good way to ensure long-term maintenance. This means that OpenSesame, despite having been around for a long time and having become key infrastructure for many researchers, is still vulnerable. If I decide to quit, or if I become less successful in obtaining research grants and therefore have less freedom to spend my time as I choose, there is a real risk that OpenSesame will stop existing. Many important open-source projects are in the same position. They are strongly reliant on the goodwill and availability of one person or a very small group of people.

As OpenSesame continues to evolve, how do you balance the need for backward compatibility with the introduction of new features and improvements? How do you ensure that updates do not compromise the reproducibility and reusability of experiments created with earlier versions of the software?

I use so-called semantic versioning, which means that you can tell from the version of the software to what extent it is compatible with previous versions. For example, as I’m writing this, the latest version is 4.0.29.

• The first number (4) is the major version. Each major version is neither forward nor backward-compatible with the previous major version. This means that it’s not guaranteed that you can open experiments built in 4 using 3 (forward compatibility) and also not that you can open experiments built in 3 using 4 (backward compatibility).

• The second version (0) is the minor version. This means that 4.1 will be able to open all experiments made with 4.0 (but not the other way around).

• The third version (29) is the maintenance version. This means that 4.0.29 is the thirtieth (counting from 0) release in the 4.0 series, each time addressing a bug or translation issue without changing any functionality.

When you are maintaining a large program for a long time, this kind of systematicity and predictability is necessary. Users understandably hate it when they find out that something has broken after an update.

As someone who has led a successful open-source project, what advice would you give to other researchers or developers who are working on creating and maintaining their own research software? What would you do differently if you were to start again?

I would first ask myself if I would be willing to maintain the software at least for the medium term, which I mean for say the next five years. If not, then I would see it as a project aimed at personal use, and maybe for a few colleagues, and just let it evolve depending on what my own needs are at the moment. I have these projects too, for example, a parser of electroencephalographic (EEG) and eye-tracking data that is mainly intended to be used by my own research group.

But if yes, then I would focus on making sure that the project stays manageable. That it doesn’t spiral out of control or become too messy. This means creating documentation, and also a system to generate documentation automatically insofar as possible, so that you don’t have to manually update things too often. This means systematically and professionally releasing the software, for example by creating a pip-installable package in the case of a Python project. This means establishing style guidelines for the code, so that it stays readable. And this means thinking about whether and how users can find support. (You’ll find quickly that email is not a sustainable solution!)

But I would not overthink it. The future is too unpredictable for that. When it comes to OpenSesame, I don’t think I would do anything differently. Not because I had the perfect plan, but because OpenSesame was not the product of a plan, but rather of a gradual evolution. And that’s also ok.

What impact do you hope OpenSesame will have on research practices in social sciences?

I see OpenSesame as a tool that should work as smoothly as possible for its users. It’s up to users how they want to use the software. So I don’t see the software as a way to steer research practices in one direction or another. I do have ideas about how research should be conducted, of course. But OpenSesame is not a vehicle through which I try to make that happen. I mainly hope that OpenSesame will make research easier and cheaper to conduct.