In short: AI agents are heralding a new revolution with the ability to delegate tasks to large language models (LLMs).
AI agents are heralding a new revolution with the ability to delegate tasks to large language models (LLMs). Imagine being able to ask ChatGPT not only to give you the schedule for the train you want to take, but also—and most importantly—to make the reservation for you!
As soon as we delegate the ability to act to the LLM, we see a large number of new and very interesting use cases emerge. In the case of automated software testing, in particular, we can envision the development of agents that can not only write tests but also—and most importantly—execute them completely autonomously.
However, challenges remain, and AI agents still suffer from numerous shortcomings. One of the main hurdles lies in their ability to take action, which requires answering two questions: “What to do?” and “How to do it?”
In fact, you first need to know what to do—that is, to have an abstract mental image of the action to be performed.
For example, if the goal is to book a train ticket, you must first search the schedule to find the right train. Once you know what to do (search for a train), you need to know how to do it—that is, understand the app you want to use and navigate its graphical user interface.
Continuing with our example, you need to know how to use the search bar and enter the information needed to find your train.
LLMs are now very good at providing answers to the question “What should I do?” In fact, if you ask ChatGPT the following question: “I’m on the SNCFconnect website; what do I need to do to book a train from Bordeaux to Nantes on March 15?”, it will reply:
To answer the question “How do I do this?”, we need to be much more specific and explain how users can interact with the app automatically.
In the field of web applications, agents actually operate by using technical frameworks such as Playwright, for example. If you ask ChatGPT the following question: “How do I use Playwright to perform step 2 (search for your route)?”, it will return some Playwright code, but the provided parameters won’t work!
In fact, the CSS selector [data-testid="origin-input"] does not exist in the web application.
The challenge of “How to Do It?” is becoming known as “grounding.” It involves starting with an abstract description—the answer to the question “What to Do?”—and providing the technical details needed to interact concretely with the application’s graphical user interface. The main challenge lies in identifying the components with which to interact.
In recent years, several potential solutions have been considered in the field of web applications. The first involved providing the web application’s DOM with a description of the element to be interacted with.
In our example, we provide the DOM of the sncf-connect page and ask the LLM to give us the CSS selector for the search bar. This approach yields disappointing results, mainly due to the complexity of the DOM.
A second approach involved identifying all the interactive elements on the page, constructing their CSS selectors, and asking the LLM which of these selectors corresponds to the element we want to interact with. This approach did not yield better results.
A second set of approaches has recently been developed that makes use of screenshots. The SOM (Set Of Marks) approach involves generating a screenshot of the website, but graphically circling all interactive elements and labeling them with numbered graphical markers.
Once this is done, you can ask the LLM for the graphic label number of the area surrounding the element you want to interact with. This approach yields very interesting results, but some elements are difficult to identify (menus, small buttons, etc.).
Finally, some LLMs are trained to return the X and Y coordinates of objects. We can then ask them to locate objects. Here again, the results are encouraging.
In any case, the grounding problem has not yet been solved, and there are many studies proposing innovative solutions. In the field of testing, this echoes the problem of a website’s testability. Indeed, we can expect the first results to come from applications that are easy to test.
If that is the case, we could soon see the arrival of agents to assist with testing, which would allow for more thorough testing of the applications.
Mr Suricate, France's leading provider of no-code automated testing
The no-code SaaS solution Mr Suricate a wide range of automated tests, allowing you to take control of your acceptance testing and provide your users with an optimal experience.
Take control of your applications and detect bugs in real-time across your websites, applications, and APIs by automatically replicating your user journeys at regular intervals.
Further Reading
FAQ
What is an AI agent used for testing?
It is an AI capable of handling tests more autonomously: proposing, executing, and adapting test scenarios based on an understanding of the intent.
What is the grounding challenge?
The challenge lies in grounding the AI’s actions in the concrete reality of the application, so that it acts accurately rather than in an approximate manner. Without grounding, the agent misses its target.
Are AI agents replacing testers?
No. They speed up the design and maintenance of tests, but they remain under human supervision, particularly to verify the appropriateness of the actions taken.
What is the purpose of AI agents in the test?
Reduce the effort required to create and maintain scenarios, especially when dealing with rapidly evolving applications.
How does Mr Suricate AI in the test?
By leveraging AI to enhance reliability and reduce maintenance—without adding complexity—for both business teams and key accounts.
