November 30, 2008

test-first VS writing a test harness

There is a common understanding that code written using a test-first approach (as in TDD-style) ends up being more maintainable than code that has been harnessed with tests later on. I would justify that understanding as follows:

TDD simplifies original design. This one is easy. By driving the design through small unit tests, we never write code that is unnecessary and experienced TDDers know to make the tests pass with the simplest thing that works. Further, because we gain immediately from writing testable code, the code usually ends up loosely coupled with no implicit dependencies and few explicit dependencies.
Another thing is that by writing the code that exercises the implementation, we design both sides of the API, client code and implementation at the same time. I blogged about client code as design some time ago.

The tests once failed for lack of functionality. To make sure the test is bug-free -- and actually testing the upcoming functionality -- it is imperative to make it fail before making it pass. If you're testing first, that's easy. If you're writing a test harness, you might have to comment out a bit of code to get the red bar, but then will it still compile? Will it still be coherent at runtime? A test harness is forever dubious.

TDD gives you fine-grained coverage. With very little overhead, TDD gives you fine-grained coverage since you're writing the simplest test that brings you forward, all the time. In the words of Gerard Meszaros, this gives you better Defect Localization. When writing a test harness, we usually emphasize the higher level functionality and forget about the smaller 'units' of work. These smaller units might be hard to test because they are implicit, hidden/encapsulated, or even unfathomable.

Test/Specification mismatch. The tests are the requirements for the code. The BDD people have taken this to the next level, but it remains true with TDD. Test code is the most useful documentation to a fellow developer: it's working, up to date, terse, and it serves as an example of client code. When developing TDD-style, the tests become the developers' understanding of the stories -- the ultimate reference.
If writing tests later on, we need to match the functional requirements with the actual codebase, like pieces of a puzzle. Except there will be mismatches, where the developers were not really clear about the requirements. What are we to do then? For this reason alone, test harnessing a big codebase is just really, really hard.

Interestingly, most of the benefits derived from test-first development are gone even if you write the tests right after new code.