AI as a Decision-Making Tool – An Interview with Marc Hage Chahine

In a nutshell: As a Methods and Tools Expert at Sogeti, Marc Hage Chahine has always worked in testing and has wanted to share his experiences. This guide covers: Why aren’t we making progress?, What benefits can AI bring to test automation?, Personally, how much do you trust AI?, So, what would be the ideal context for it to work?.

In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The world of testing is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find the answers to these questions in our interview with Marc Hage Chahine.

As a methods and tools expert at Sogeti, Marc Hage Chahine has always worked in testing and has wanted to share his experiences. He is fortunate to be able todo so in his daily work, as well as by writing articles for the “Taverne du testeur” blog and magazines such as Programmez!. Marc Hage Chahine also enjoys helping to organize major testing events such as STLS (in Sophia Antipolis) and JFTL.

Test automation is a rapidly evolving market. According to the latest estimates from Research and Markets (2021)*, its revenue could reach $49 billion by 2026. In your opinion, what does the future hold for test automation? 

Test automation has been touted as the near future for over twenty years. People keep talking about it as a replacement for manual testing. But it’s clear that it doesn’t replace manual testing and never will, and that implementing it is complex. If we look at the latest CFTL surveys, test automation has not made progress over the past two years, even though we are working in agile environments where automation is increasingly necessary.

Why aren't we making any progress? 

For a long time, automation remained highly technical, involving code and development. But in recent years, we’ve started to see no-code or script-free tools emerge alongside Keyword-Driven Testing*, with RobotFramework® as its leading example. This is the “zero generation” of testing for non-technical users. We also haveGherkin®, which performs automation based on the same principle. There are other tools that are becoming increasingly mature, accessible, and functional. But even though they’re accessible, they’re still fairly technical tools designed around code and scripting. 

Since then, new tools have been coming out:Keysight® with Eggplant,Tosca® from Tricentis, and the latest versionof UFT®, where they’re trying to add AI, blocks, and modules so that people can use them more easily. There’s also a lot of image recognition, which allows for code-agnostic testing and enables a single script to run tests on different phones (iOS, Android), tablets, PCs, and even embedded systems. The next evolution will be AI. We’re seeing new developments with tools likeGravity® from Smartesting, which analyzes logs to suggest test flows and automate them. 

Keep in mind, just because all these tools exist doesn’t mean they’ll all be used. There are cost barriers—the question is whether or not to make that investment. The teams in charge of product development also need to have the right policies and strategies in place for testing and know how to manage quality. It’s important to remember: to have good automated tests, you first and foremost need good tests. Teams need to recognize the need for automation—automation shouldn’t rely solely on the tester. 

Speaking of AI, how do you see it fitting into test automation?

For me, AI is there to facilitate decision-making and action. It’s nothing more than automation that goes beyond simply automating execution. It should help us make choices, but AI won’t do everything. We need data and information, which means we need people behind the scenes. With AI, we can beat Go or chess champions, but chess AI won’t beat a Go champion. It’s very specific, so it’s in specific cases that it will outperform us and help us move forward. There are many use cases for AI, particularly in test prioritization, which leads to selection. For example, if we have 500 regression tests, which ones should we run first to find the most bugs? But in the end, we’ll only be able to run 400 or 450 of them, so we need to determine which ones we won’t run.

What benefits can AI bring to test automation? 

AI can help in several ways: it can assist with analysis—particularly of failed tests—and it can help identify use cases highlighted by vendors that are failing, determine why they’re failing, and even make them work again and correct the scripts directly. AI will also assist with maintenance and upkeep by enabling better management of the test suite. For example, if there are duplicate tests, AI will suggest that we consider removing them since they serve no purpose, or it will suggest that we revise the data for certain tests, and so on. AI will also assist with test creation: we specify what we want to do, and the script is generated automatically—which is, in fact, what scriptless tools already do.

The only real limit to AI will be what we imagine it can do. The other limits will be the costs of implementation and also whether it’s worth doing, because just because we’ve imagined doing something doesn’t mean it’s actually worthwhile. For example, flying cars—we’ve been imagining them for 50 years, and we could very well build them, but in the end, they serve no purpose, they don’t add any value, and they consume too much energy. There has to be a need for them.

In your opinion, what are the potential risks of AI in terms of test automation and the testing profession? 

There are a lot of risks associated with AI. The first is a bias stemming from overconfidence: if the AI says it, the AI must be right, so I’ll do what the AI says. This is often linked to the fact that we don’t have time. AI should be a decision-support tool, but it shouldn’t be the one making the decision. If we don’t have time to think things through, review the metrics, or examine the data, we’re no longer talking about decision support—we’re talking about decision-making. And that’s the real risk with AI, given that AI isn’t perfect. It relies on data—data that, even if it’s up-to-date, relevant, and representative, can change overnight. Let’s take a very current example: the presidential election polls: Macron was at 20–25% in the polls, then a new factor emerged—the war in Ukraine—and he jumped to 30–35%, then another new factor emerged—consulting firms—and he dropped back to 25%. What’s true today isn’t necessarily true tomorrow, and AI isn’t capable of anticipating that.

Personally, how much trust do you have in AI? 

By default, I’m skeptical. I see potential, possibilities, and contexts where it will work. For example, forGravity®, we have two case studies on APIs that will be presented at the JFTL (French Software Testing Day). In that context, within that framework, it works well, so I’ll trust that, in a similar context, we’ll be able to implement a solution that will save time. Otherwise, there’s no trust until there’s feedback; by default, it doesn’t work.

So, what would be the ideal situation for this to work?

The ideal scenario involves repetitive, tedious tasks—things that aren’t necessarily difficult, but are easy and not very interesting to do. They’ll require doing a lot of easy things, but the sequence will be tiring and eventually become difficult. All of this lends itself to automation, and that’s been the case throughout human history, for that matter: we created tools because it was easier to crack a coconut with a sharpened stone. And each time, we’ve freed up more and more time. 

What did you think of this interview? Be sure to check out our other interviews with Bruno Legeard and Xavier Blanc as well. 

FAQ

In recent years, innovations in machine learning and artificial intelligence have been driving change across many industries. The world of testing is one of them. But how can AI be integrated into test automation? How can it help business teams better understand this field? What are its benefits, risks, and limitations? Find out the answers to these questions in our interview with Marc Hage Chahine. ?

Marc Hage Chahine, a methods and tools expert at Sogeti, has always worked in testing and has wanted to share his experiences. He is fortunate to be able to do so not only in his day-to-day work, but also by writing articles for the Taverne du testeur blog and magazines such as Programmez!.

Test automation is a rapidly evolving market. According to the latest estimates from Research and Markets (2021)*, its revenue could reach $49 billion by 2026. In your opinion, what does the future hold for test automation?

Test automation has been touted as the near future for more than twenty years. People keep talking about it as a replacement for manual testing. But it’s clear that it doesn’t replace manual testing and won’t replace it, and that implementing it is complex.

Why aren't we making any progress?

For a long time, automation remained a highly technical field, involving code and development. But in recent years, we’ve begun to see no-code and script-free tools emerge alongside Keyword-Driven Testing*, the leading example of which is Robot Framework®. This is the “Generation Zero” of testing for non-technical users.