Firefighter Man, I'm going to catch it for that title. Okay, let me get this out up front: this post is a hypothesis that I consider probable. That's why there's a question mark in the inflammatory title. Okay, asbestos underwear in place, let's continue.

The Hypothesis

Here's the logic flow for my hypothesis.

  1. Unit testing (and its bosom buddy automated testing) provides substantial benefits when used regularly. These benefits are well-understood.
  2. Ensuring or regularizing unit testing is the primary value proposition of Test-driven Development (TDD).
  3. A majority of vocal unit testing advocates are also TDD proponents.
  4. Developers searching for unit testing information are more likely to find references that describe unit testing as subservient to or at least included in TDD than they are to find references that are simply about unit testing.
  5. Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn't have had a problem with simpler unit testing methods.

Simple enough, really. If 1, 2, 3, and 4 then 5.

I'm not going to bother trying to verify statements 1 and 2. You might be able to quibble with 2, but it'd take some real mental gymnastics to do so.

Testing Propositions 3 & 4

Propositions 3 and 4 are key to this hypothesis being correct so let's explore them. Here's a simple test: what are the top 10 links on Google for unit testing and are the addresses they link to substantially advocates for test-first development. This test assumes that the top results from a Google search will be representative of what unit testing advocates are saying. Since Google earns money by the metric ton to make sure this is the case, I'm going to let the results stand for both 3 and 4.

To help with the analysis, I'll include two rankings for each link. Both rankings are my subjective opinion; included so that you can check my figures for yourself if you wish by seeing how much you agree with my evaluation (or if you agree at all, I suppose). Both are a scale from 0 to 5.

The first scale, Helpful, will indicate whether a developer looking to find initial unit testing information is likely to read the whole thing. A five is something I think most developers will stop to read and digest. A 2 or 3 is either too general or too specific and thus might or might not be interesting to a developer looking for initial unit testing information. Lower scores indicate links that make me question Google's ranking system.

The second scale, Test-first Focus, will indicate how focused that link is on writing unit tests before coding. A five is something that puts unit testing squarely in the realm of test-first development and includes substantial explanations on how to do so or a base assumption that testing first will be practiced by those reading. A 3 means that it contains information on testing first, but in a more neutral tone—i.e. unit tests will be written when the developer determines it is important with testing first given as a (mere) option. Lower scores indicate that testing first is mentioned in passing if at all. And yes, I am going to lump TDD into XP and other test-first methodologies because they will tend to lead to step 5 as much as formal TDD will.

Okay, here's the top 10 Google hits for unit testing as of this moment:

Link Description Helpful Test-first Focus
Unit testing - Wikipedia, the free encyclopedia This is the wikipedia entry for unit testing. As is common for wikipedia, the article is concise and well-focused. TDD is mentioned but only briefly and as a "related topic". 3 1
Effective Unit Testing A magazine article supposedly about unit testing. Third sentence: "What if I told you we are going to start with unit testing?" 2 5
Extreme Rules A page on unit testing at "you should try to create your tests first, before the code." 4 4
Six Rules of Unit Testing A developer's list of six rules for unit testing. Good post, actually. "1. Write the test first" 5 5
Chapter 13. Unit Testing A chapter from Dive Into Python. The link itself is only the introduction to the chapter, but the very next section leads with this: "Now that you've completely defined the behavior you expect from your conversion functions, you're going to do something a little unexpected: you're going to write a test suite that puts these functions through their paces and makes sure that they behave the way you want them to. You read that right: you're going to write code that tests code that you haven't written yet." 2 5
JUnit, Testing Resources for Extreme Programming JUnit is an automated unit test framework for java. I've never used it, but I understand it is the gold standard for Java unit testing frameworks and the conceptual ur-progenitor for unit testing frameworks in other environments. 3 0
Unit Test A page from WardsWiki. I've no idea what that is, but its primary focus seems to be software development. It's almost more of a forum than a wiki, though. That said, the introduction for the topic includes this: "Key here is to CodeUnitTestFirst. " 1 4
Testing, fun? Really? An old blog post (March 1st, 2001) about unit testing in Extreme Programming. "Unit tests are so important that you should write your tests before you write the code." 5 5
Unit Testing in scriptaculous wiki A wiki for the Java Script unit test framework. I didn't know they had those. Cool. 2 0
Unit Testing in BlueJ A pdf manual for unit testing in BlueJ (I've no idea what BlueJ is and don't feel like looking it up). Chapter 10: Writing tests first. This is actually a little misleading, though because the chapter is about how to test first if you want to. 1 3

Well, fancy that. Over half the entries are a 4 or 5 for their test-first focus. Of those that aren't directly test-focused, one is wikipedia and the others are unit testing frameworks (I think—that last one is weird).

I'm going to call propositions 3 and 4 proven.

So Then

Here's the question: given that 3 and 4 turn out to be true, do you think that "Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn't have had a problem with simpler unit testing methods"? Personally, I think this is likely the case.

Since the barrier to implementation is higher for TDD than it is for POUT, you have to acknowledge that more developers will be willing to spend the resources on POUT than will do so for TDD. The question is how many are we talking here? How can we find out? I find TDD a much more substantial resource hit than POUT. Thus, at least some people would be willing to spend resources on POUT that wouldn't spend them on TDD. You can double-check this by trying to imagine someone being willing to spend resources on TDD who would hesitate to spend them on POUT. Do you think there are any? I don't.

If this is a trend in the general population, then what happens when someone who would opt for POUT if they knew about it is only able to find information for TDD? It'd be only natural if they skipped unit testing altogether.

TDD folks gripe about projects with no unit testing. Non-TDD folks resist and resent being badgered about implementing a methodology they aren't convinced will justify its cost. I wonder if we aren't locked in an unnecessary struggle between poles with each side providing ammunition to the other.

I wonder what would happen if the TDD folk started with encouraging people to unit test regularly. After all, TDD folks would have to admit that having some unit tests is better than no unit tests. How about starting your testing discussions with POUT? Or how about defaulting to POUT when you run into resistance for TDD and push for more later if you can? How badly do you want regular unit testing? Badly enough to let go of presumptive TDD discourse? I've noticed that TDD advocates never discuss unit testing outside of a TDD framework. Since many of the most popular developers fit this category, I wonder if we aren't unwittingly encouraging a TDD or nothing effect in the wild.

I wonder what would happen if every TDD advocate went on the record that POUT is way better than no unit testing (with examples and/or encouragement and without going on to propound the vast superiority of TDD). Will you TDD advocates reading this accept the challenge from me to do so?

And here's an interesting question you can address: how much of the advantages of TDD do you think can be achieved with just good design and timely unit testing? 90%? 50%? 20%?

Technorati Tags: ,,