Firstly, neither is sufficient on it’s own. They have different goals, different strengths and weaknesses and different constraints.
Manual Testing = Exploratory Testing = Testing = Information Gathering = Learning
Testing is the means by which we increase what we know about something. Every interaction we have with a system allows us to learn something new about it. To that end, we are all testers, we all have a certain skill-set and the variance in these skill-sets allows us all to test in a different way and learn different things about the system.
Testing lends itself to a situation in which we want to know more about something. Perhaps we have developed something new, we want to know how this new function performs in a variety of different scenarios. If we see unexpected behaviour, we want to be able to investigate and understand why this behaviour is apparent. We may have existing functions which we don’t know much about, or maybe they are about to be used in a new and different way to how they were used previously. Testing allows us to learn how these functions may perform and behave in the new situation.
Automated Testing = Automation Scripts = Scripting = Checking
On the face of it ‘Automated Testing’ performs a similar job to Manual Testing. Some actions are performed and then we decide if the system under testing is working based on some outputs. But, this is a misleadingly simplistic view. Automation cannot diagnose the cause of unexpected behaviour, it cannot adapt to new changes and adjust its’ reaction accordingly. It cannot use initiative to decide to investigate further or ignore something as a ‘false negative’.
Herein lies the key difference with ‘real’ testing. It does not allow us to learn anything new. It applies what we already know and validates whether or not it is still true. In other words it provides ‘change detection’.
So when an automation suite runs it can tell us one of two things:
- “One or more of my checks failed, therefore something in the system has changed”
- “All of my checks passed, therefore I could not detect any change”
One of the main weaknesses of automation is that neither of these outcomes tells us beyond doubt that the system is working satisfactorily.
Running the automation and not detecting change can give us a certain level of confidence that the system is working properly - though without testing, we cannot know for sure.Having automation run and it detect change does not necessarily mean that the system is broken. But it does highlight an area of the system which we may wish to spend additional time testing, to learn about the functions which appear to have changed.