The discussion spawned from my TDD post has been interesting to me. I've enjoyed the comments by Phil on his post and by him and others on my own. I'm particularly flattered that one of the authors of the original study dropped by. I disagree with some of his points, but this isn't where I want to address them. Instead, I want to examine the discussion itself because this is definitely something that we as a software development community could be better at.
First, I want to emphasize something that I could have made clearer in my original post. The original study was extremely well designed and I have no issues with how it was conducted or how it was structured. I described the report in my prior post but neglected to give my opinion of the setup. Let me correct that now: this is an exceptionally well-designed, well-executed, and well-written study. And I'll point out that my criticisms were not with the data collection or design. If you go back over what I said, you'll see that my concerns are with presentation and interpretation.
As I said, I enjoyed most of the comments posted both on my post and on Phil's. That said, here's some advice for those expressing opinions online.
First off, if you are going to criticize something, it'd probably be best if you actually read it. Or at the very least, read what is said about it very carefully. I can tell that many of those supporting or criticizing the report hadn't bothered to get to know the study at all because of little misrepresentations they made (like assuming that two teams from a group of 24 must be split evenly—they weren't). Here's the thing, if you make little errors of fact, it becomes hard to take the rest of your points seriously.
Second, if you are going to criticize the way a study is put together, it'd help if you know what you are talking about. Take sample size, for example. While 24 seems like a small sample size to end up with, there are very detailed and specific methods to adjust for sample size in statistical mathematics. This fact is at the heart of the authors' use of the term "statistically significant". In essence (and strongly backed by decades of research) if a bias is strong enough, sample sizes don't have to be very big to be representative enough to be useful. A little cross-discipline humility goes a long way here.
Now, I have no problem with someone who might say "that seems like an awful small sample size" because that allows you to express your doubt but also leaves room for further enlightenment and avoids making an unsupportable assertion of failure. The thing is that this study wasn't put together by amateurs. This is an academic paper presented to the IEEE. Like all good academic papers, it includes an entire section titled "Threats to Validity". Reading it, you'll notice that sample size is not listed in this section. Asking (yourself or in general) why a sample size of 24 isn't considered a threat to validity is a good idea and potentially interesting in a geeky kind of way.
Third, the force of your conviction doesn't ameliorate the burden of making your point clearly and rationally if you want to convince someone else. Too often, advocates will substitute emotion for reason when discussing their Great New Thing™. That's a perfectly human thing to do. It is also perfectly human to reject overwrought arguments out of hand. Seriously, you do your cause no good if you cannot be bothered to make your case in clear and rational terms.
Finally, if you don't know people you can simultaneously respect and disagree with, the problem isn't that people who disagree with you are all stupid jerks. The limits of a system isn't in the brilliance of those arguing for it, it's the brilliance of those arguing against it. Untested theories and unquestioned understanding are as useful as untested code. Personally, for any given topic, I want the best arguments that I can find. It's not enough to examine both sides of some new thing, I want the very best arguments that both sides have to make.
Which is why I appreciate the efforts of people like the authors of this study. And why I appreciate guys like Phil who take the time to discuss their theories and to explain and expound them when challenged. None of this stuff is a no-brainer. If it were, people wouldn't pay us to do it.