Posted by Jacob Stevens on June 21, 2010
My esteemed colleague Jon Bach just made the very good point in his “Automated Baseball?” entry that automation is, ultimately, a test tool. It doesn’t replace testing, but assists and empowers testing.
So here’s another example of a test tool in the guise of something else: code compilers.
Compilers check for syntax and nomenclature. Correct but inapplicable syntax and nomenclature can be accepted and compiled, and later caught by testing, but it’s largely a task testers just don’t have to do anymore. Implications on test and quality still exist, and are, to me, fascinating.
Compilers also perform type checking. Type checking checks for proper handling and usage of objects according to their data types (string, int). If an object is assigned a data type of string, the compiler will check to make sure all the code treats it like a string, calls to the object only call attributes a string has, no disallowed implicit type casting is attempted, etc.
I should clarify that this is only for static programming languages. Static-typed languages like the C family of languages (C, C++, C#, Objective-C) perform their type-checking at compile time. Dynamically-typedlanguages (Python, Ruby, SmallTalk, Lisp) perform type checking at run-time. Some, like Perl,have both (built-in types are static, but Perl supports dynamic type-checking for user-defined types).
How is this relevant to testing? They have different implications. Type-checking at compile time is kind of “conservative:” it won’t let you compile some code that could result in a run-time error, but may not, depending on how the objects are used, how errors are handled, or depending on user input. This is information only available at run-time, and so in lieu of contextual certainty, compilers play it safe and throw an error.
Dynamically-typed languages like Python are arguably more powerful by letting you do things a conservative compiler wouldn’t, allowing you to self-mitigate against run-time errors with more advanced exception handling.
So compile-time checking is generally safer and conducive to more stable code. As testers, we think that’s good! But run-time checkingconstitutes much more testing (it makes unit tests actually useful!), and allows for far more complex risks, that require more complex test coverage, because of the nature of run-time errors compared to compile-time.As testers, job security ain’t so bad either, is it?
So it’s a “risk-for-power” tradeoff, and for many developers, it’s worth it. A widely cited practice with Python is EAFP, which means, “It’s easier to ask forgiveness than permission.” It’s mostly a performance and error handling posterity thing, but it’s got implications on risk, quality and testing.
Which is good or bad, depending on your perspective.Risk-abating test engineers may consider it a bit reckless and not think the tradeoff is worth it. From the perspective of quality engineers, who want to help build great software and contribute quality to development, it may simply pose more of a personal challenge.
But regardless of your perspective, the usage and role of the compiler, as really a test tool, reminds me of Jon’s point, that whatever the tool, we’re still just testing. Contributing quality to a development effort.
And with those objectives in mind, as a quality engineer I see the differences and implications of compile-time and run-time type checking, and I wonder, why not both?
Any test engineer can testify, compilers don’t find all the bugs. Testing for correct and stable program operation, can find a much wider range of bugs, than compile-time type checking.On the other hand, it’s impossible to be as comprehensive and fast as compile-time type-checking at finding the bugs that can be found with a type-checking compiler.
So why not both? Static-typed languages still benefit from testing, after compilation. Wouldn’t dynamically-typed languages benefit from a compiler with full type-checking, at the very least to alert the developers to all the potential issues, that they could then choose to ignore and handle themselves, when/where it’s beneficial? And otherwise allow type-checking to be performed at run-time, where testing can identify what needs to be fixed?
We use automation to supplement test: we let it show us where to investigate, where an issue may be, and then we make the judgment call. I would think dynamically-typed languages would benefit from the same thing. There’s bound to be some veteran Python programmers out there, who’ll read this and enlighten me on why compile-time type checking doesn’t work for them. But as I understand it right now, a valuable tool is at your disposal, so why not use it?
