I learned something today. I may not know how to run even the simplest scripted test case.
I’ll show you.
This one is for the File/Open dialog of Internet Explorer:
- On a clean XP machine, launch Internet Explorer
- Go to the File Menu, select Open
- Enter the word “TEST” into the text field
- Click OK
Verify that “https://test/” appears in the URL text field.
Ok. Let’s look at the word “clean” in step 1. What does “clean” mean? If it means I should start with a fresh install, what about upgrades that the OS will want to run when I connect to the internet?
Let’s look at the word “launch.” How does the test case engineer – the person who wrote this – want me to launch it? Double-click the .exe or shortcut or command line? Does it matter? No? How do you know? Furthermore, as a tester, I could be thinking “is that a flaw in the design or is he leaving me free to choose any old way?”
What about which version of IE?
What does the word “go” mean? If it’s a means to invoke the function, does that mean via the keyboard or mouse?
What about the word “enter” – can I cut and paste into the field? After running this test case several times, I may choose this method – will it alter the test?
What about the word “TEST”? Is it meant to be capital letters? What about the quotes?
You can see there’s a lot of room for thinking here.
Ok, so you say to the tester:
“No thinking allowed… besides, it should be obvious.”
I would argue it’s not, simply because of the problem of being human. The tester could even hate the notion of exploring, loving to have their tests written for them, but the simple fact is, three such testers running this simple script still have the same problem to solve: how do I run this? In answering that question, each may find three different answers for themselves, running it three different ways.
“It doesn’t matter,” some might say. “Any way is fine, it’s all about the spirit of the test – the Verify statement.”
Alrighty then, the tester might confirm the “TEST” text is in the right place, but if the application crashes, is that a pass or a fail?
It’s obvious – it’s a bug to file.
Well, not really.
It could be that the programmer already knows about it and it’s unrelated to the population of the URL bar, which is really what they wanted to know from the test.
In short, if I find that a test step is vague, I might need to ask a question.
In asking the question, I am seeking context.
In seeking context, I am thinking and adjusting.
In thinking and adjusting, it’s exploratory.
“If it’s not exploratory, it’s avoidatory,” my brother James often says in a Johnnie Cochran voice, maybe for the times when you need to avoid any peripheral actions or problems when you run a test. This can be good if your mission is to only find bugs in one feature that will be demonstrated during a trade show. Any bugs outside of that feature are not important, so do not explore! As in the case above, if the word “TEST” appears, that’s all we care about.
Michael Bolton calls this kind of testing “ceremonial.” He did a talk recently at the Pacific Northwest Software Quality Conference where he made the following analogy: When the Queen of England reviews the troops in formation, it is ceremonial. She does not inspect them in detail like a sergeant would, saying “Is that dirt on your cheek, solider?!? Drop and give me 20!”
Anyway, some people in our biz want to compare the efficacy of scripted testing against exploratory testing. I used to be one of them, but study and experience with testing over the years has made me bow out of this silly idea.
It’s not because I believe that ET has been more effective for me in finding better bugs quickly. It’s not because I believe that test cases are an easily prescribed way of accounting for your testing.
It’s because I have not yet seen evidence that a scripted test case can be run by a human the same way every single time. On the flip side, it’s also because I believe exploratory testing can include scripts. Because this line is blurred for me, I don’t know what it is I’d be comparing in a test-off between scripted and exploratory.
If “scripted” means “written down in advance,” that could mean that when I’m exploring, (which many think of as “unscripted”), I am doing scripted testing when I use a script or a procedure to frame my exploration. Rightly so, I can have a model of the test or the feature, a charter, a mission statement, a user role I’m supposed to play – yet still be what Cem Kaner calls “brain engaged”: alert and attune to emerging context.
I think we all do this. We think as we test.
What I’m asserting here is that the EXTENT to which we are ALLOWED to think is the extent to which the test is exploratory.
If our tests are written for us in advance and if we are told not to think, (which could well be the case if we are trying to repro the exact steps from a customer, developer, or our own bug regression case) then I suggest that even still, we can not define it as scripted or exploratory, only a *degree* of being “more scripted” or “more exploratory.”
Here’s something that helps me in those cases, called the Tester Freedom Scale:
After my brother and I came up with this, it’s easier for me to talk about “scripted” vs. “exploratory,” because I can talk in degrees of freedom. More freedom is more exploratory, less freedom is more pre-scripted.
Maybe that’s why doctors write these things called “prescriptions” – instructions written down in advance of going to the pharmacy to give the procedure – the exact dose and the duration – without having the pharmacist needing to think or interpret.
But aren’t we glad that the pharmacist is human, and can choose to call when they are confused because of the doctor’s bad handwriting?
