Code coverage tools enable a user to see which lines of code they’re executing and which lines they’re not for a given project. During testing this can be helpful to find any gaps where code isn’t being executed, and get a feel for what sections are being tested more or less. However there’s a tendency to think about code coverage as being equivalent with test coverage, particularly since it can be reduced to a definite percentage, which isn’t the case.
If 83 percent of the code is being executed, doesn’t that mean that 83 percent of the product has been tested? Not really, a section of code may have been executed once, but it may also be called again by untested portions of the code and fail in those cases. For example, code that processes user data may count on the caller to validate entries, and behave perfectly in some cases, but fail in others when the caller fails to validate properly. If you only test the working cases, you’ll still achieve coverage on the code, but it won’t be fully tested.
Also, testing in different environments can be equally important without adding anything to the code coverage percentage. Does your code working on multiple platforms? at different permission levels? with different configurations? This sort of thinking may be commonplace in testing, but it’s important to not lose sight of it when looking at coverage results.
So why use code coverage tools at all?
While it’s true that a high amount of code coverage may not garauntee a high amount of test coverage, a low amount of code coverage is almost always a red flag that something’s wrong. Either test is missing something or their may be some fundamental design issues.
Furthermore looking at which lines are not being covered may uncover a feature or a branch that hasn’t been tested, laying a great foundation for further areas that test can explore. For example in a recent project there was a branch of code that only executed when using a feature running as a non-privileged user on Vista.
Alternatively coverage tools can sniff out dead code, and help keep the code-base up to date and more maintainable, which helps the project as whole.
When used properly, code coverage tools are an excellent asset for test and the rest of the team. It’s important to recognize what they provide and what they don’t, and not to fall into the trap of summarily thinking that the amount of code coverage is the amount of test coverage. However if that can be avoided, then the benefits are there for the taking.
