In brief: A regression test verifies that a code evolution has not 'rolled back' a functionality, meaning it hasn't broken what was previously working. It is also referred to as a non-regression test (NRT). Essential for every update, it ensures the stability of a website or application. This guide explains what a regression is, the types of regression tests, when and why to perform them, their limitations, and the benefits of automating them.
When aiming to validate the quality of a web or mobile application, numerous tests are available. Among these, regression testing stands out as one of the most essential. But what exactly is it? Let's explain.
What is a Regression?
Fundamentally, the word 'regression' denotes 'an evolution that reverts to an earlier stage'. In software development, a regression occurs whenever a code change impacts existing code. This can result from a bug fix, a software update, or the addition of a new feature.
For instance, if a newly implemented feature disrupts the behavior of previously existing functionalities, a regression occurs, as a bug has been introduced. To prevent this, technical teams implement specific tests, known as regression tests or non-regression tests.
What is a Regression Test (or Non-Regression Test)?
According to ISTQB's definition, a regression test involves retesting a previously tested program after a modification, to ensure that no defects have been introduced or uncovered in unmodified parts of the software as a result of the changes made.
In summary, a regression test serves to verify that modifications made to software, a website, or a mobile application, such as adding a new feature or an update, have not adversely affected previously existing functionalities. Consider the example of a bicycle: if a wheel is replaced, a regression test would involve checking that, despite the wheel change, the brakes still function correctly (though it's not necessary to verify that the bicycle pump still attaches properly).
What is the difference between a regression test and a non-regression test? In reality, there is no difference; they refer to the exact same concept. Both terms are used interchangeably. ISTQB, for instance, prefers the term 'regression test'. However, many professionals in the field also use the term 'non-regression test', particularly in France. It's a matter of preference, and it's true that for those unfamiliar with the domain, this can lead to confusion. What can one expect? The French are complex beings who enjoy wordplay!
Regression Tests and Functional Tests
Functional tests verify that software (website, mobile application, API, etc.) operates in accordance with the specifications determined by the client beforehand. On an e-commerce site, this would involve, for example, verifying that account login, adding items to the cart, or selecting a delivery method does not lead to bugs and that no discrepancies are observed compared to the previously provided data.
Regression tests, on the other hand, are performed when a new code version is released, to ensure that it has not introduced errors into the rest of the software. They can target both functional and non-functional aspects, such as performance. For example, if an update causes the website or application's response time to be longer than usual, this can be considered a regression.
What are the different types of regression tests?
Regression or non-regression tests can be executed in several ways, depending on the specific requirement or the company's resources.
Corrective regression tests, for instance, reuse existing tests, provided no significant changes have been made to the product.
Full regression tests involve retesting all elements of the product. This allows for verification of all modifications made since the beginning of the project.
Selective regression tests, conversely, involve choosing specific tests from a suite to inspect only the impacted parts of the code.
Progressive regression tests involve creating new tests when established tests are no longer relevant, for instance, when product features are modified.
Partial regression tests are conducted when various modules are under development and are about to be merged with the main codebase.
Unit regression tests, for their part, are used to test code individually, without considering other elements.
When to Perform Regression Tests?
Regression tests can be executed at all levels of the test plan, and it is advisable to perform them as regularly as possible, whenever a modification or update is introduced, ideally from the earliest stages of product design.
However, non-regression tests are generally performed whenever there is:
>> a code correction is implemented to resolve anomalies
>> a new feature is added
>> an existing feature is modified
>> an update is performed in the environment (e.g., data)
>> source code optimization is carried out
Why Perform Regression Tests?
The primary reason for conducting regression tests, and one that should be central to all other types of testing, is that it allows for ensuring software quality (website, mobile application). By delivering a quality product, user experience is enhanced, thereby improving the company's image.
Furthermore, performing regression tests helps to mitigate risks associated with updating applications, websites, etc. This is crucial because, in addition to potentially causing bugs that hinder the use of certain functionalities (which can impact the company's revenue and image), it can also lead to security vulnerabilities. In such cases, the company's credibility and user trust are at stake.
Beyond considerations of quality, risk, and brand image, the benefit of regularly executing regression tests is to save time and money, as correcting a bug in production is always more complex and costly. Moreover, these are the most suitable tests for automation, which also saves time and money, a topic we will elaborate on later.
Finally, in the context of developing mobile applications or SaaS software, which are continuously updated to meet client requirements, regression tests become even more essential.
What are the limitations of regression testing?
Regression tests are highly time-consuming and repetitive, requiring significant effort. While the majority should be automated, some test cases still necessitate manual execution. Furthermore, certain complex functionalities demand intricate test scenarios, which can also delay execution and, consequently, delivery times. In summary, due to time and budget constraints, not all regression tests can be performed, making it crucial to carefully prioritize which ones are essential to conduct.
What about test automation?
As mentioned earlier, non-regression tests are particularly well-suited for automation, and they are often the tests recommended for priority automation. Why? Because they are repetitive and time-consuming. They are executed with every new feature deployment and software update; performing them all manually is incredibly time-consuming, and the repetitive nature of many simple tasks can become tedious and challenging. This is precisely where the automation of regression test cases becomes highly valuable.
Finally, test automation enables the earliest possible identification of potential regressions and, crucially, helps to maintain delivery cadences, thereby improving ROI. However, it is not necessary to automate everything; it is generally advisable to prioritize test cases that frequently exhibit defects, test cases that verify essential and critical product functionalities, and test cases that cover functionalities that have undergone numerous and recent modifications.
FAQ
What is Regression Testing?
This is a test that verifies a code modification has not degraded an existing functionality. The word 'regression' denotes a return to a previous state: the test ensures that the software does not decline in quality after a change.
What is the difference between regression testing and non-regression testing?
In practice, both terms refer to the same objective: ensuring that an evolution has not introduced any breakage. 'Non-regression' (NRT) is the most common terminology today.
Should regression tests be automated?
Yes, this is the ideal use case for automation: these are repetitive tests that must be re-executed with each update. Automating them saves considerable time and enhances the reliability of each deployment without requiring team involvement.
Further Reading
To see the solution in action, check out our non-regression tests.
