The science of testing software

Professor of Software Engineering and Testing at Open Universiteit and at the Universitat Politècnica de València in Spain Tanja Vos has always been way ahead of the troops when it comes to her field of research. ‘Software testing has long been viewed as a simple task. Even now, when our lives are dictated by software applications, I still have to convince people about the importance and fundamental challenges of testing software.’

Where and when did you get intrigued by software testing as a field of research?

‘Originally, I wanted to study mathematics. But when I heard that there might be slightly more women in computer science, I decided to enroll in that program. There turned out to be a whopping three female students in my year… Nevertheless, I never regretted my choice, since I immediately liked the fact that in computer science, you learn how to make software that can do incredible things.
Slowly but surely, during my studies I migrated towards software engineering, and did my PhD in the field of formal methods, a mathematically rigorous approach to prove the correctness of software. When I moved to Spain after obtaining my PhD from Utrecht University, I took that experience with me to a technical institute where we helped software developing companies with problems their customers encountered with their software. There I learnt that formal methods are great when you need to evaluate critical software, but are highly impractical for most Spanish small software companies, which, at that time, typically developed enterprise resource planning software. In such systems, of course it is problematic when things go wrong, but no one will die. What’s more, the software they develop typically is updated or even rigorously changed in relatively short timespans. As a result, these companies do not have the time nor the resources to use formal methods to guarantee the quality of their software. They need testing methods that are way more flexible.
I immediately saw ample opportunities for myself to make meaningful contributions to this field.’

Ever since you entered this field, you have manned the barricades to advocate the urgency of software testing and emphasized the need for scientific research in this field. Why is that?

‘Everyone thinks testing software is easy, that everybody can do it, but that is far from true. To test a piece of software in the right way, you need a lot of different skills. First and foremost, you need to be able to think out of the box and imagine all kinds of scenario’s the programmer did not foresee when writing the code. You need to be able to explore, experiment, think critically, be aware of your assumptions and tacit knowledge, i.e. knowledge that involves insights, intuition, or know-how that a person has developed over time but may not even consciously recognize or be able to verbalize. That is also the reason why it is so hard to automate testing, and although with AI we are getting a step ahead we still need the human in the loop.
Knowing this, it is surprising that testing is an underrated topic, both in research and education. In my over twenty years of teaching, I have often struggled with students who consider testing boring and have no real interest in learning how to do it well. It is my belief that this is because we are teaching it wrong. Because of that, I consider it my mission to train my students to be aware of possible risks and hence test well at every phase of software development, from writing the first line of code to full deployment.
What’s more, with AI getting into the daily lives of almost everybody, I argue that we need to establish testing skills in society as a whole. As humans, we need to be able to evaluate the outcomes of AI software and for that, everybody needs testing skills!. That is why I am currently setting up an experiment with primary and secondary schools with the aim to promote critical testing literacy in school children.’

What is your current research about?

‘The backbone of my research is TESTAR, a tool that enables the automated system testing of desktop, web and mobile applications at the level of the graphical user interface (GUI). We developed this tool some ten years ago within the context of an EU-project named FITTEST (Future Internet Testing).
When testing applications at the GUI level, the most important question is: What can go wrong when a user is using the software, and what are the possible consequences? Usually, this approach leads to so-called scripted testing, where your test follows a predefined script and evaluates the functionality of the system. However, the problem with scripted testing is that applications currently developed change so frequently that is almost impossible to keep up with scripted testing. As soon as the software changes, you need to change your script as well and this leads to a costly maintenance problem. In our research, we try to turn this around. TESTAR is a scriptless way of testing the software, which incrementally adds layers of intelligent testing. When there are no scripts, these do not need to be maintained.
For example, in a current NWO-project called AUTOLINK we are exploring the possibility to turn TESTAR into a tool that can learn for itself how and what to test when it comes to applications with a graphical user interface. We are automatically deducing what possible actions a user can take, and to what new situations those lead. In the AUTOLINK project, we are researching how to make this system more intelligent, and automatically test realistic user stories. To this end, we are collaborating with Utrecht University, where there is ample knowledge about user stories and how to capture them.’

You are dividing your time between Spain and the Netherlands, in a group that is located in both countries. On top of that, you are active in the Dutch software engineering community as the chair of VERSEN. Why do you invest your scarce time in this?

‘As a service to the community, would be the short answer. It is crucial that there is an association that puts the spot on software engineering and the importance of investing in the basics of this field. People tend to forget that topics like cyber security and artificial intelligence that are currently very popular, are built on software. I want to help create awareness of the importance of funding these basics to establish a strong foundation under those kinds of special interest topics.
It is my ambition to extend VERSEN across the national borders and join forces with like-minded organizations across Europe to establish a long-term European agenda in the field of software engineering. In the Horizon Europe program, for example, in 2024 there has been exactly one call on fundamentals of software engineering, for which everyone in the field went in competition with each other. In the end, only four proposals were granted. That is a major waste of time and money, and we should do better for our field.’