Whenever I run a test that passes, I never stop wondering if it will pass when the customer does the same thing.
Up until today, I thought that was just me being a worrier and that I needed to seek counseling about it. But a discussion today on software-testing@yahoogroups.com helped convince me otherwise.
Whenever a test passes, it’s only an *indication* that the software CAN meet a requirement or expectation. It is no guarantee that it WILL.
That implies that tests that pass are really just educated opinions about a product’s capability, not an assurance that it will meet customer expectations.
Sure, they may call it Quality Assurance, but I’ve always been uneasy about that term. I can’t assure anything when it comes to software because I can think of so many contexts that might change or affect software when the customer goes to run it.
So forget the idea that passed tests say “the software WILL do x, y, z.”
I can’t assure such a statement any more than I can assure I’ll be in a good mood tomorrow.
I prefer to say “the software CAN do x, y, z in this context.”
I’m not trying to be bulletproof or get out of being responsible to the concerns of stakeholders, I’m just trying to set their expectations about testing. That way, when it ships and they come to me with a bug and say “Didn’t you test this?” I can point to my testing and remind them of what CAN really means: “I had reason to believe it is capable of x, y, z because I ran a test on this context and got results that I considered to be consistent with my idea of a customer expectation around those features.”
I know… it’s not a sexy sound bite, but it’s the scientific, rational answer – appropriate for a person (a tester) whose job it is to be rational and scientific.
Maybe that’s why people say “I CAN and I WILL” when they want to emphasize both their capability and assurance about something. I just think it’s worth it to separate those two powerful words when we report our testing to stakeholders who may not see a difference.
