In short: A regression test verifies that a code change has not “set back” a feature—that is, broken something that was working. These are also known as non-regression tests (NRTs). Essential for every update, they ensure the stability of a website or application. This guide explains what regression is, the types of regression tests, when and why to run them, their limitations, and the benefits of automating them.
When you want to verify the quality of your web or mobile app, there are a wide variety of tests available. And one of the most essential is none other than regression testing. But what exactly is it? Here’s an explanation.
What is regression?
Essentially, the word “regression” refers to “a change that reverts to an earlier state.” In computer science, a regression occurs whenever a change to the code affects the existing code. This can be caused by a bug fix, a software update, or the addition of a new feature.
For example, when a new feature is implemented, if it disrupts the behavior of previously existing features, then a regression has occurred because a bug has been introduced. To prevent this, technical teams set up specific tests, known as regression tests or non-regression tests.
What is a regression (or non-regression) test?
According to the ISTQB definition, regression testing involves testing a program that has already been tested, after a change has been made, to ensure that no defects have been introduced or discovered in the unmodified parts of the software as a result of the changes made.
In summary, a regression test is used to verify that changes made to the software, website, or mobile app—such as the addition of a new feature or an update— have not affected previously existing functionality. Let’s take the example of a bicycle: if you change the wheel, a regression test would involve verifying that, despite the wheel change, the brakes still work (but it’s not necessary to check that the bike pump attaches properly).
What’s the difference between regression testing and non-regression testing? In reality, there isn’t one—they’re exactly the same thing. Both terms are used. The ISTQB, for example, prefers the term “regression testing.” But many people in the field also use the term “non-regression testing,” at least in France. It’s all a matter of choice, and it’s true that if you’re not familiar with the field, it can be confusing. What can you do? The French are complex people who love to play with words!
Regression Testing and Functional Testing
Functional testing verifies that a piece of software (website, mobile app, API, etc.) works in accordance with the specifications determined by the client beforehand. On an e-commerce site, for example, this involves verifying that logging into an account, adding items to the cart, or selecting a shipping method does not cause any bugs and that there are no discrepancies with the data provided previously.
Regression tests, on the other hand, are performed when a new version of the code is released, to verify that it has not caused any errors in the rest of the software. They can focus on both functional and non-functional aspects, such as performance. For example, if an update results in a longer-than-usual response time for the website or application, this is considered a regression.
What are the different types of regression tests?
Regression or non-regression tests can be performed in several ways, depending on the specific need or the company's resources.
Corrective regression tests, for example, reuse existing tests, provided that no significant changes have been made to the product.
Comprehensive regression testing involves retesting all components of the product. This allows us to verify all the changes that have been made since the beginning.
Selective regression testing, on the other hand, allows you to choose specific tests from a set in order to inspect only those parts of the code that have been affected.
Incremental regression testing involves creating new tests when existing tests are no longer useful, such as when product features are modified.
Partial regression tests are performed when different modules are under development and are about to be merged into the main branch of the code.
Unit regression tests, on the other hand, are used to test the code individually, without taking other elements into account.
When should regression tests be performed?
Regression tests can be performed at all levels of the test plan, and it is recommended that they be conducted as regularly as possible— as soon as a change or update is made—starting as early as the product design phase.
But generally, non-regression tests are performed whenever there is:
>> a fix made to the code to resolve issues
>> a new feature that is added
>> a change to an existing feature
>> an update made to the environment (e.g., data)
>> an optimization of the source code
Why run regression tests?
The primary reason for conducting regression testing—and one that should take precedence over all other types of testing—is that it helps ensure the quality of the software (website, mobile app). And by delivering a high-quality product, we also improve the user experience and, consequently, the company’s reputation.
Furthermore, conducting regression testing helps reduce the risks associated with updating applications, websites, and so on. This is important because, in addition to potentially causing bugs that prevent certain features from working (which can impact the company’s revenue and reputation), it can also lead to security vulnerabilities. And that’s where the company’s credibility is at stake, as well as users’ trust in it.
In addition to ensuring quality, managing risk, and maintaining a positive image, the benefit of running regression tests regularly is that it saves time and money, since it’s always more complex and costly to fix a bug in production. Not to mention that these are the most suitable tests to automate, and automation also saves time and money—but we’ll come back to that later.
Finally, in the context of developing mobile apps or SaaS software—which are continuously updated to meet customer requirements—regression testing is all the more necessary.
What are the limitations of regression testing?
Regression tests are very time-consuming and repetitive, and they take a long time to complete. While most of them should be automated, some test cases must be performed manually. Furthermore, certain complex features require complex test scenarios, which can also delay execution and, consequently, delivery. In short, due to time and budget constraints, not all regression tests can be run, so it’s important to carefully select which ones should be prioritized.
What abouttest automation?
As we mentioned earlier, there are real benefits to automatingnon-regression tests, and in fact, these are the tests we often recommend automating first. Why? Because they’re repetitive and time-consuming. They’re run every time a new feature is deployed or a software update is released, and doing them all manually takes time—a lot of time—not to mention that it involves performing many simple tasks, and the repetitive nature of the process can become tedious and difficult. This is where automating regression test cases really makes sense.
Finally, test automation helps identify potential regressions as early as possible and, above all, enables teams to maintain delivery schedules and thereby improve ROI. However, it is not necessary to automate everything, and it is generally recommended to prioritize test cases that reveal frequent defects, test cases that verify essential and critical product features, and test cases that verify features that have undergone numerous and recent changes.
FAQ
What is a regression test?
This is a test that verifies that a code change has not degraded an existing feature. The term “regression” refers to a return to a previous state: the test ensures that the software does not experience a decline in quality after a change.
What is the difference between regression testing and non-regression testing?
In practice, both terms refer to the same goal: ensuring that a change hasn’t broken anything. “No regression” (TNR) is the most common term used today.
Should regression testing be automated?
Yes, this is the ideal use case for automation: these are repetitive tests that need to be run with every update. Automating them saves a considerable amount of time and ensures the reliability of each deployment without tying up the teams.