Regression testing can never be fully automated
In a software testing context. A regression is when a piece of software inadvertently returns to a previous state. Possibly though the introduction of a new feature or a bug fixes gone wrong.
Something which should never happen.
I have covered what regression testing is here. So feel free to read up on the subject if you’d like!
In that article. I stated that the aim of 100% of regression tests being automated should never be the goal. And in this blog post, I want to expand on that a little further.
Automating a subset of your regression tests can bring a lot of advantages to your testing strategy. From being able to free up a testing team from running long and large amounts of repetitive tasks. Allowing for increased accuracy of the tests which are run. Not to mention the distinct advantage of running your regression suite as soon as a new build of your software is released. Enabling faster feedback on the quality of your new software build.
It is important to be aware that neither approach is ‘better’ than the other. Nor should one be chosen, while eliminating the other.
Automation in software testing is just testing, but with benefits
Each method has its own unique advantages. Such as enabling you to gain key insights that the other does not provide. But they also come with their own disadvantages as well.
| Automated Regression Testing | Human Regression Testing | |
| Run tests quickly and repeatedly in a short amount of time | Yes | No |
| Report is something is ‘right’ | Yes | Yes |
| Explore if something really is ‘right’ | No | Yes |
In regard to answering which tests you should automate on a broad scale. I’d refer you to this post. But to answer which tests should be automated in a regression suite. It boils down to a number of different factors, from whether the selected tests cover the basic workflow of your application. If the tests are important enough to be included (like testing security features of an application). Or whether they are tests are reliable enough and have caught bugs in the past.
However, this is not a yes or no answer, and the question of automating needs querying further.
Can it be automated?
The route from your home to the office is one you probably know better than the back of your hand. Every single day, you get in your car and drive the same route. But I think we can all agree that getting behind the wheel isn’t a great experience all the time.
Often there are road works to navigate around, traffic to consider and other road users too. Can’t that process be automated for us? We know all the steps involved after all.
But although we know the process for driving your route. There are several variables involved that are harder for us to predict.
- What will the road conditions be?
- Will there be obstacles to navigate?
- How will we react if something unforeseen happens?
The example just given is representative (if it were a test case), of something that should not necessarily be automated.
It would be incredibly difficult to program an automated script to handle the numerous amounts of possibilities. And even if it were possible to do so. It would probably take far too long to make it worth the time that would be saved.
So for now at least. This is one task that is better suited to humans who can think, react to and calculate all the various variables that a machine can not currently handle.
Although that is not to say self-driving isn’t possible as Tesla shows us.
Can we automate it?
Once you have identified that a regression test is a good candidate for being automated. A vital question that you should ask is if you can even do it in the first place.
With various tools and frameworks available for testers to automate the layers of an application, from the UI down to the unit level. Do you even have the skill to do it within your current team?
A friendly developer colleague might be able to help in this situation, or they had the foresight to design the application with automation in mind.
However, if nobody on your team has the skills currently. Then I would highly suggest in leaving it for a human tester to pick up.
Should we automate it?
This question is probably the most important when it comes to selecting tests to be automated.
As I explained in this post. Automating something takes time. It’s not something that can be whipped together quickly and needs to be properly designed to avoid flakiness.
For that time to spent well and maximize your ROI in test automation. You need to select the best test cases that make the most sense to be automated.
And to do that, you should be looking at those tests that are going to be repeated multiple times over the length of your software development project. There is no point in automating something that is only going to be run once and then never again.
Because in my opinion. Once a test has been run and passed. If it is added to a regression suite. It becomes a check. If a test is run and then not run again. It’s still a test. But a test that serves no future purpose.
So why would you automate it?
If you need a regression test suite created for your software project. Get in contact today.
