In short: Bruno Legeard is an expert in test automation who has been working for several years on using AI to streamline software testing.
In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The testing industry is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find out the answers to these questions in our interview with Bruno Legeard.
Bruno Legeard is an expert in test automation who has been working for several years on how AI can facilitate software testing. As a professor of software engineering at the University of Franche-Comté, he supervises several doctoral dissertations on this topic. He is also one of the co-founders of Smartesting, which develops AI-based test automation tools, and he has contributed to the new AI Testing certification through the ISTQB.
You gave a presentation on using execution traces to optimize automated regression testing with AI. Could you tell us a little more about that?
Once we recognized the need for test automation, the question became: “How do we do it?” Current practices that rely on coding automated scripts present challenges today because they require a significant amount of maintenance effort. This is where AI and new technologies come into play—to reduce this maintenance effort, on the one hand, and to ensure that automated tests cover key user flows in production, on the other. This is the topic we addressed, and for which we proposed an AI-based solution that analyzes software usage traces to supplement automated test coverage. Testing the user flows that matter—by analyzing what users actually do.
Aside from this maintenance effort, what other benefits can AI bring to test automation?
A range of AI-based techniques is beginning to emerge in several areas related to test automation. For example, when an automated test is a traditional test—such as an end-to-end web test—it simulates user behavior, which requires the ability to interact with interface objects. And often, failed tests result from a change to these graphical objects. Machine learning makes it possible to find the correct object and automatically modify the script to fix the locator— that is , the location of the graphical object that must be activated during the test. But that’s just one aspect.
The second issue is the relevance of automated tests— that is , ensuring that key user flows are consistently covered by automated tests. And there is a third important issue, which is the prioritization of tests during execution. When you have end-to-end tests that ensure user flows are functioning properly, execution can take time. One solution to this problem is to prioritize the execution of automated tests based on insights from previous results and predictions of which anomalies will be detected.
So the idea is to run scenarios that will more often result in a "KO" to quickly assign a status to, for example, an error correction?
Absolutely. The magic of AI and machine learning is that it’s not about rules defined by humans. It’s the model’s learning—algorithms learning from other algorithms—that enables us to make predictions with a high degree of reliability. Let’s say we have 300 tests to run; 17 of them will fail and reveal anomalies. With prediction, those 17 tests will then be among the first 20 to be run. If we have time to run all 300 tests in 3 minutes, we won’t need it, but that’s often not the case in reality. The reality is that the 300 tests take 4–5 hours to run, so we have a strong interest in having reliable prioritization. And AI, in our work, is a supervised learning technique—it involves learning from historical data, the context in which the test is run, its duration, the changes that have been made, and so on. AI should be viewed as a day-to-day facilitator for testers.
On the contrary, in your opinion, what are the limitations of AI in test automation?
In a few years, it seems pretty clear to me that we’ll be able to develop intelligent, autonomous testing systems. There are developments in the lab starting to emerge around self-adaptive testing. The number one issue is regression testing—tests that are tedious to maintain and run. I’d bet that within 5 to 10 years, we’ll have this type of robot. We’ll have capabilities that will automatically replace the creation of tests by testers in certain environments. This also means that tasks like defining test profiles and applying domain knowledge—which are somewhat repetitive—will be partially automated. However, testers provide significant added value across all quality engineering activities, and the implementation of intelligent, autonomous test bots will serve to support and enhance their work.
AI is a fairly complex topic. In your opinion, what challenges might it pose for test automation?
When using and implementing machine learning techniques based on data sources, if the data sources are unreliable, then the AI results will not be reliable. If we seek to predict defect rates or risks associated with different software components based on development history, this historical data must have been archived, maintained, and be reliable. This is also an issue; it’s not enough to simply say, “I just need to use this AI engine to get a certain result,” such as predicting a component’s error rate. If you don’t have the right data upstream, the component will never produce any useful results. Testers, in fact, have a role to play in ensuring that the data used by the AI is of high quality.
*End-to-end testing: Testing a complete integrated system to verify that all integrated components function in the final environment across targeted user flows.
*Screenplay: A step-by-step guide to testing one or more features in a web or mobile application.
What did you think of this interview? Be sure to check out our other interviews with Marc Hage Chahine and Xavier Blanc as well!
FAQ
In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The world of testing is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find out the answers to these questions in our interview with Bruno Legeard. ?
Bruno Legeard is an expert in test automation who has been working for several years on how AI can facilitate software testing. As a professor of software engineering at the University of Franche-Comté, he supervises several doctoral dissertations on this topic.
You gave a presentation on using execution traces to optimize automated regression testing with AI. Could you tell us a little more about that?
Once we recognized the need for test automation, the question became: “How do we do it?” Current practices that rely on writing automated scripts present challenges today because they require a significant amount of maintenance effort.
Aside from this maintenance effort, what other benefits can AI bring to test automation?
A range of AI-based techniques is beginning to emerge in several areas related to test automation.
Further Reading
