How to handle testing everything
Software testers are often under pressure to ensure that the testing we do covers as many of the inputs of an application as possible.
But exhaustive testing is not possible due to multiple reasons from time restrictions, to complexities in software and variables in the environment.
There is a ceiling to testers abilities that can be a lot lower than the number of features, paths and sequence of operations in software projects.
This can often lead to a tug of war occurring, with one side being testing too much. The other, too little. A fine line we have to navigate if we are going to be able to say that we are confident that a certain feature is working.
In this article, I will share a couple of tips to ease the task of balancing these two possible scenarios. Methods that allow you to rely on the confidence of your passing test cases. As well as enabling you to be more effective and efficient with your testing.
First, I like to think of the time spent on testing as a tangible object. One that can be spent to further the success of the project. Or if misspent, a waste of resources that doesn’t really help in achieving your testing goals.
Test cases are an expense
To do this, I find it helpful to think of software testers time like the money in a bank account.
Every time you execute a test, some money goes out. And when you find a bug, you receive some money in your account.
The goal in testing is to get as much money as possible for the tests that we execute. We can use automation as a tool to help us in executing some tests for us. But writing the tests, again, drains our bank’s funds.
A single test equals funds being spent.
Testing every input would bankrupt you
Let’s give the example that we are asked to test a form which makes use of an input field which accepts the numbers from 0-10.
The initial reaction might be to test all the valid inputs from 0-10 and see if any results in a bug.
If this were a larger application however with hundreds of possible combinations and variables. It would take up most of the testers time to input, execute and record the results of each possible test.
That would be a very expensive task.
Also, if a valid input results in no bug being found. I would be surprised if an input value in the same range resulted in an error being uncovered.
For example, if the input of “2” resulted in a test passing, then I would be confident that “8” would produce the same result.
Instead of testing every single combination. It would be better to make use of test techniques like boundary value analysis and equivalence partitioning to create test cases that have the best chance of catching errors.
This ensures that all parts of the input in which logic is invoked gets exercised by our tests. And the repetitive task of filling out the form with every possible combination isn’t required.
Of course, there are other possible tests that should be added for minus numbers, non-numerical characters and so on. But I’ve only focused on the ‘happy path’ for simplicity.
There can be multiple ways to reach the same result
The other day, I was trying to download the source of a webpage using Python with the help of the requests library. I soon ran into difficulties when I was only able to download a portion of the page’s data.
I couldn’t understand it. The page was rendering fully when I loaded it in my browser. Why was my script only downloading part of the page?
After scratching my head for a while and debugging the issue. I opened up Chrome DevTools and discovered that the reason for the source of the page to be incomplete was the page was being rendered by JavaScript. Causing the page source not to be fully displayed if the visiting request didn’t have JavaScript enabled.
To open up Chrome DevTools. You have a few options depending on the platform you are on. You can open it by the menu, More Tools and then Developer Options, you can right-click on a page, and go to the Inspect option. Or you have the option of using a couple of keyboard shortcuts.
If I were a tester at Google, testing a new version of Chrome and needed to test various features within the DevTools. I absolutely would need to test each one of those input options to ensure they yielded the expected result.
Not doing so is kinda like having a car that is advertised with four doors. But only testing one and thinking that because that one works. The other ones will too.
But when it comes to testing features inside DevTools. Such as the ability to enable certain options and disable certain features.
Would I need to write tests for each input method?
Randomise your test inputs
I would suggest in this instance that while it is tempting to test everything. The fact that is too expensive and too time-consuming to carry out all possible combinations. Makes doing so impossible to achieve.
Instead I would advocate the use of randomised inputs for your test cases. Ensuring even distribution of each input type. Enabling you to be confident that if a user tried a different input method. The fact that it has passed in a separate test, gives a very high chance that it will pass again in different circumstances.
Conclusion
Testing everything in software is impossible. We can’t control that.
But we can control the types of testing we do, in order to target as many variations as possible.
Test coverage should always outnumber tests executed.
Was this helpful? Do you have any other ways of navigating the ‘test everything’ problem? Let me know!
