In a nutshell: Xavier Blanc is a software quality specialist. This guide covers: What risks might AI pose to the world of testing? And, just how much do you trust AI?
In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The world of testing is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find out the answers to these questions in our interview with Xavier Blanc.
Xavier Blanc is a specialist in software quality. As director of the Bordeaux Computer Science Research Laboratory (LaBRI), he co-founded ProMyze, a startup that helps define and share best practices in software development, thereby improving team performance. He is also a professor at the University of Bordeaux, specializing in software engineering.
AI is a hot topic right now, and it’s starting to make its way into the world of testing. Personally, how do you see AI fitting into test automation?
AI can help with automation, in particular, by not leaving the automation engineer to work alone: how can we assist the automation engineer in their work to help them produce higher-quality tests? Because ultimately, when you write an automated test, you’re repeating operations—and that’s where AI has proven to be very strong. When we start repeating tasks—when data is already available and we’re doing things we’ve done before, or that have been done by others—that’s where AI can help us improve quality . We’ll take an application and ask the AI to analyze it and help us identify the most relevant tests.
There’s a second point regarding diagnostics. We have our test suite, we know how to automate it, it runs very smoothly, and yet sometimes there’s still a diagnostic issue that’s difficult to resolve. The test tells us it’s failing—it’s red, so there’s a bug—but it’s not actually a bug; it’s something context-dependent: maybe the network isn’t working properly, or there’s latency. And we end up wasting time on the diagnosis. We have experience, but it’s so complicated that intuition alone isn’t enough—and that’s where AI can quickly deliver significant benefits.
What challenges can AI pose in the area of test automation?
AI will have a hard time being very context-aware. The applications we’re testing are similar, but they all have their own specific characteristics, and these characteristics mean that the choices that work for some applications don’t work for others. Creating an AI that will solve all the world’s problems isn’t even a realistic goal. Instead, what we’ll be aiming for is an AI capable of adapting to the context. So it will need to learn very quickly, discern the specific characteristics of that context, and be highly relevant. And that’s where we’ll face enormous challenges in the coming months or years: identifying that context and knowing how to make decisions that may be radically different, rather than relying on a dominant average—which is where AI currently excels: it has plenty of data points and knows how to find the curve that connects them all, but now it will have to be able to filter out the points that don’t make sense in the context of a specific application.
What would be the ideal context for AI to be integrated into test automation in this case?
I’d like to see AI built right into the apps. We often have massive amounts of data and spend time analyzing it. We run the algorithms and end up with an AI system, but it lacks a bit of responsiveness—and having the AI directly within the app will allow it to be highly responsive to user behavior. User behavior changes overnight; it’s almost impossible to predict, and we’re always a step behind. We have the data, we clean it up, and then we run the algorithms.
I think we’re going to see AI built right into applications that will minimize the time it takes for this feedback loop. AI will be able to make recommendations: “Users are doing this today; you should look into these kinds of tests,” and so on. We’re already seeing this in cybersecurity. They’re constantly monitoring systems, which allows them to detect new attacks. They also use collaborative AI: they have multiple sites and are trying to pool their knowledge. We could imagine this kind of thing happening between different applications, allowing us to realize that a user’s behavior has changed or, conversely, that there are anomalies. And if there’s AI embedded within the application, we should get feedback back to the developers very quickly, enabling them to anticipate bug fixes and be much more responsive.
In your opinion, how could AI help business teams—who don't have a technical background—with test automation?
If we have AI in the application, it would need to be capable of speaking two languages: the language of the business users and the language of everyone working on the application’s development. If the AI is intelligent enough, we could imagine it understanding users better and reaching out to the business users to tell them exactly what the users are doing. For an e-commerce site, this would make it possible to know what users are buying. Business users should then be able to respond by suggesting this or that based on the AI’s data. Similarly, the AI should alert developers to where there’s a problem—for example, on the screen, etc.
This would also allow industry professionals to conduct business tests and A/B testing. Currently, we set metrics and monitor them, but we could also determine whether it’s worth it or not. And if I do that, what would the AI predict? Will it be worthwhile or not? We could also imagine the same thing from a technical perspective: if I do that, will I introduce new bugs or not?
What risks might AI pose to the testing industry?
To return to the metaphor of points, AI is very good at finding the curve that passes through all those points, but if we ask it what the next point is, it will give us the next point on the curve it has drawn. The problem is that if we don’t choose the right points—if we include too many—the AI will lead us down paths that are off course.
And a second problem that can arise isthat there may be points that aren’t on the curve but are still worth considering. We’ll also have to recognize that there are times when we may need to turn off the AI and take risks. And the winners will be those who have managed to take risks that go against the AI’s predictions.
So, there are these two pitfalls: having an AI that isn’t very smart and sends us in the wrong direction, and an AI that’s smart but not particularly bold—one that won’t suggest taking a path that isn’t heading in the right direction, even though it would be a huge success. AI calculates averages, so that’s the risk—that we get a little too stuck in a rut.
Speaking of which, how much trust do you place in AI?
I have absolute confidence in AI, if we consider that AI is the algorithm that finds the curve that passes through all the points. That’s not a very honest answer, but I don’t want to blame AI just because we’ve given it bad data. So AI learns—if I feed it unintelligent data, it will learn nonsense. The challenge is to provide it with intelligent data from which it can learn. We also have algorithms that are beginning to understand how to structure information so that AI learns quickly, but as for asking AI to sort through the data itself—deciding what to use and what to discard— I don’t think it’s up to the AI algorithm to do that sorting—it happens upstream, and that’s where we’ll need to be a bit more rigorous about how we feed in the data. So yes, I’m confident, and I think we’ll achieve even higher confidence rates if we’re able to provide the right data points. Do I trust the data points we provide? Much less so—but for me, that’s no longer the AI’s responsibility.
Can you tell us about the work you're currently doing on testing and AI?
We’re doing a lot of work on what users do through the web app, and we’re starting to have quite a few metrics that let us measure their behavior. Based on that, an AI should be able to tell us who our users are. There are plenty of technologies known for performing these kinds of classifications: there’s the mainstream user, the very rare user, and so on. The idea is that we should be able to provide automated test suites automatically, saying: “Here you go—if you want to test what the average user does, just run this; if you want to test users who are a bit more trend-conscious, here’s what you should do instead.” The challenge here is to take all the actions that humans perform on the apps and try to see if we can classify them, if we can categorize these actions and ask the AI to generate different patterns. Once we have these behavioral models, we’ll be able to ask ourselves: won’t these models help us test the app more effectively? This ties back to everything I’ve said: currently, these behavioral models are created after the fact, and we’d really like to have them in real time—to see if they change from day to day—by integrating AI directly into the app.
What did you think of this interview? Be sure to check out our other interviews with Marc Hage Chahine and Bruno Legeard as well !
FAQ
In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The testing industry is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find out the answers to these questions in our interview with Xavier Blanc. ?
Xavier Blanc is a specialist in software quality. As director of the Bordeaux Computer Science Research Laboratory (LaBRI), he co-founded ProMyze, a startup that helps define and share best practices in software development, thereby improving team performance. He is also a professor at the University of Bordeaux, specializing in software engineering.
AI is a hot topic right now, and it’s starting to make its way into the world of testing. Personally, how do you see AI fitting into test automation?
In particular, AI can support automation by not leaving the automation engineer to work alone: how can we assist the automation engineer in their work to help them produce higher-quality tests? Because, ultimately, when you write an automated test, you’re repeating operations—and that’s where AI has proven to be particularly strong.
What challenges can AI pose in the area of test automation?
AI will struggle to be very context-aware. The applications we’re testing are similar, but they all have their own specific characteristics, and those characteristics mean that the choices that apply to some applications don’t apply to others. Creating an AI that will solve all of Earth’s problems isn’t even a goal.
