Are there any limitations to test automation?

Isn’t it great when one solution works for any problem that you might have?

When I was a kid. I was always curious why older people were always trying to boss me around, telling me where to go and what to do. Then I discovered the response ‘Why?’. And while it helped to satisfy my curiosity for a while. I soon discovered that it had the side effect of being quite (understandably) annoying and therefore, it’s usefulness waned over time.

Or how ten years ago if a company wanted to make the tech product better. The go-to solution would be ‘Add Bluetooth to it!”.

Kinda reminds me of that scene from Big Bang Theory.

In the software testing domain. Many believe that if you want to make your testing ‘better’ and less manually intensive. You can just throw automation at the problem… But is this thinking correct?

While automation is a wonderful thing and can save you time and resources in your software testing efforts. Understanding the limitations of the technology in a software testing context. Will help you define your software testing strategy and understand where and you we can use automation to help achieve the results you desire.

First, let’s define what automation is, and is not

Words really do matter here and while test automation may bring up images of a robot running test procedures on an application with no human interaction. That is not what test automation is.

Test automation can not test. It can only verify what a human has asked of it previously.

I realise this may be a controversial statement. But it is important for you to understand what test automation can do for you, and what is outside its jurisdiction.

It may even be useful to you to refer to test automation as automated verification, automated checking, or even automation in software testing. Though test automation is widely used and understood. So I’ll be sticking to the term.

Test automation is a support tool, ideal for repeated testing, and is definitely not a magic bullet solution.

OK, but what can it do?

Short answer, A lot.

With test automation, you are only really limited by three things:

• Your programming experience and how well you can put into practical use the skills you have to reach your desired goal.
• Your knowledge of the tool you are using. Either a coded solution like Selenium or Cypress. Or a codeless Test Automation solution such as TestProject which even allows you to collaborate with other developers who can code (even if you can’t).
• The time and budget of the project you are part of (though, this one applies to the overall testing that you are doing).

However, those are only a sample of the limits that automation brings. Below, I will identify the ones that are important to consider before investing a lot of time and resources into something that may not unlock the solutions you are seeking.

Once a test is written… it doesn’t mean that it is ‘done’

While we can bake into our test code features that will cope with minor changes to the UI of an application, for example. It is likely that over time, your software will grow with additional features, extra steps and added complexity. Being able to write a test once and then forget about it is not something that is workable right now. Unlike a human tester who can work out changes and modifications and the actions they should be taking. Automation scripts can only do what a human has written.

Therefore, automation is never truly ‘done’. It constantly needs to be maintained, modified and evaluated overtime.

Writing automated test is a time sink

A big decision you need to weigh up before adopting automation is whether you have the time and resources to sink into it. Following on from the last point, the automation strategies that are the most robust and effective. Are the ones you have the time to invest the needed hours into building up.

That’s not to say you can’t get some significant results by writing small automation scripts here or there when you have a few moments. Test Automation does not just encompass browser, API or unit tests, for example. But any activity that automates any task associated with the job of software testing.

Writing a script to generate test data? Congratulations, you’re using test automation!

But the take-away is the bigger projects, the longer the time that you should be investing.

Checking is testing. But so can a human

The final limitation that I alluded to above is that while automation is perfect for those repeatable regression type tests. It still can’t test like a user.

It can’t think like a human and find issues with a websites UX for example. Or decide like a real-user and do things that are unexpected. It can only do what you have told it to do.

Automation is great not only for saving time, providing valuable feedback, or uncovering issues that can be hard for a human tester to uncover such as performance issues, or problems with the site reliability under extreme conditions. I remember when load testing used to be getting the entire test team to log onto a website at the same time.

But just don’t think of automation as the solution to all your problems. Research all your options. Discover if what you want to do is feasible. And understand that with test automation. You only get out what you put in.

I hope this blog post was useful to you.

Reach out to me on LinkedIn or Twitter if you have any questions. Or contact me using my contact form.

PS: I also send out a weekly newsletter which contains links to articles I’m reading, the books I’m enjoying. And other interesting content. Sign-up here if you’re interested. It’s free!

• Category: Software Testing