Why I hate test coverage tools

Because they could lie if we abuse them. I have been in one project where people count acceptance tests (written in FIT) so the test coverage always looks pretty high when the unit tests coverage are actually poor. But the figure gave people there false confidence that their code is in good shape.

In another project, a test had only two lines of code, but it actually ran about 5 seconds and touched a lot of code underneath. So by this test alone, it improved the test coverage a lot, looks pretty good eh? Then wait until it failed sometimes. People had absolutely no clue on why it failed and where it caused the failure. and then we could spent one hour to just pinpoint where the problem is. For me, this test is simply just not a test and should be replaced by a bunch of more fine-grained unit tests.

The test coverage number is just a number. It helps when the number is low so that you know you have serious problems in your project. But when the number is high, it doesn’t mean anything. We could still do something with it, such as browsing through the nice colored reports to see where we failed to test and if we should fix that. But we should never trust we are in good shape just because the number is high.

To make our test coverage more meaningful, we should remember that TDD is a tool to help us designing our code, to improve the code quality. If we feel awkward in writing a test, then we know there is something wrong in our code and we should refactor that piece, instead of just writing a bad test or just be off.

~ by Yi Wen on November 19, 2007.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: