User testing - is it worth it, and should we tell anyone if it's not?

I was sent an interesting article today, written in 1998 by Jacobsen, Hertzum, and John. It's hardly brand new, but it addresses something that doesn't seem to be addressed often elsewhere.

Their thesis, based on an experiment they carried out, is that the number of evaluators you have of the evidence collected from a user test is (at least) as important as the number of users. That is, more evaluators will find many more issues from exactly the same evidence, and the way they classify the issues they find is very varied. Evaluating, they suggest, is a very subjective process.

Taking into account other evidence that calls into question the classic '5 users = 80% of issues' equation (one study found issues being reported at the same rate at the 16th user), is user testing a valid, cost-effective thing to be doing? Will we really gain wisdom this way? And, if not, what are the consequences of telling those decision-makers not sold on usability that the most high-profile method is dogged with difficulty?

From looking at the evidence, it seems to me that there a few factors still supporting user testing:

1. Nielsen's quote that "The most striking truth . . . is that zero users give zero insights."
2. Most of the problems reported are down to loose definitions.
3. Evidence still suggests that properly organised and controlled user testing yields great results.

So the challenge seems to be:

• define the study properly
• choose your users carefully
• brief your evaluators comprehensively (and use more than 1)

If we do these things, I think user testing still has a central important role to play in improving user experience.

Jacobsen, Hertzum, and John's paper
Jakob Nielsen on 5 users
CHI 2003, a panel discussion on the number-of-users problem