this blog is girtby.net

Software That Goes Clunk

In the world of the manufactured consumer item, there has long been a tradition of techniques which convey an impression of underlying quality to the prospective purchaser. For example, it is never going to hurt sales if you display the words “Made in Germany” prominently on your item’s packaging. And I’ve heard that disproportionate engineering resources are typically expended on a car’s doors, in order to get just the right “clunk” to impress potential customers on the showroom floor.

The point is that you, as a mere user of the widget, can’t easily inspect its inner workings to determine the quality of engineering and manufacture, and nor do you typically have the expertise to do so. So you need to rely on the information provided by the vendor. Perhaps it plays on national stereotypes, on technobabble (hello, cosmetics industry!), luxurious packaging (hello, Apple!), or many other techniques.

With software — particularly web-hosted software — it is not so easy. An astute observer will examine the interface of the software in question and look for consistency to established standards, prevalence of modes, and so forth. A lay observer will go on aesthetics, and probably not much more.

As the market for software becomes more and more sophisticated, what are the indicators going to be for quality? “Designed in California”? Or are lay people going to look for some more substantive indicators?

I don’t know but I know what I’d really like it to be. And I know what I’ll be looking more and more closely for, in future.

Made With Unit Testing

Once you’ve gotten the unit testing religion there really is no going back. It fundamentally changes — for the better — the way that you write software.

It can also make you a lot more intolerant towards regression bugs. That’s my excuse anyway for my recent dummy spit over wordpress’ broken comments feed. This is exactly the sort of problem that unit testing can catch. So why wasn’t it? And what assurances do I have that it wont happen again? I am not a wordpress developer, and I don’t care to be one either, but as a user I want to have better confidence in the software I use.

By the way, the fact that I was running an unreleased version of wordpress when I encountered the feed validation bug, is completely beside the point. Regularly run unit tests ensure that even when trunk is broken, it doesn’t remain so for very long. As it turned out, the problem that I encountered did make it into the 2.7 release of wordpress, before being addressed. (Although, as far as I’m concerned, the bug still isn’t fixed, because there’s no unit test to prevent it reoccurring).

Even if you don’t know anything about unit testing, you should be able to factor it into a technology decision. Software package A is unit tested, and B isn’t. Which do you pick, if all else is equal?

Other Types Of Testing?

It might be argued that other forms of testing are an acceptable substitute for unit testing. To which I would say: bullshit. Or perhaps I might say “you are mistaken”, in a more professional forum. Unit tests are by definition automated, and hence a lot more foolproof. Automated testing at higher levels of abstraction is of course possible, but from experience it is far less effective. And manual testing is just not fun.

Alternatively, it might be argued that a more comprehensive indicator of quality throughout the development process is an ISO 9000 accreditation. To which I say: you’ve got to be fucking kidding me. (And let’s defer that discussion to another time).

As an experiment, pick your favourite piece of open source software. Go to the website and work out how to run the unit tests. Check out the latest source code from the repository and see if the unit tests run as advertised. If the unit tests are broken, or (worse) missing, maybe you should start looking elsewhere?

As another experiment, pick your favourite piece of closed source software (if you have one). Ask the vendor whether they have a unit test suite, and if so, what percentage of the code does it cover. Again, depending on the response, you might want to think about switching…

… to software that goes “clunk”.

Just because software is unit tested doesn't mean it's unit tested well. Just because software is covered doesn't mean it's covered well. This is sort of the exact problem with ISO9000 - It's more about intent, and less about a check-box. You can pay lip service to any sort of metric. Just look at Agitar!

There is no empirical evidence that unit testing is a better software quality technique when compared with other quality-inducing tools and practices (code reviews, test-after, etc). To my recollection, given current evidence, the winner of that fight would be one of the many types of formal code review.

I invite you to skim through all the posts for 2008 but particularly the summaries. :-) And it only gets better.

Yeah, your language could do that too… but does it? We’ve got your test religion right here.

At least some people agree with your view that "Made with Unit Testing" is a key quality indicator.

Check out the Universal Feed Parser.

They spend a total of 12 words on their home page describing the component before giving the key links and launching into example code on how to use it.

Five of those words describe what it does. Two describe the technology. Two describe the license. The other three are "3000 unit tests."

(Of course they could be 3000 worthless unit tests, but they certainly give the impression that they are serious about it.)