Panels CHI 2003: NEW HORIZONS

The "Magic Number 5": Is It Enough for Web Testing?

Carol Barnum Southern Polytechnic State University, 1100 S. Marietta Pky, Marietta, GA 30060, USA
Nigel Bevan Serco Usability Services, 22 Hand Court, London, WC1V 6JF, UK. [email protected]
Gilbert Cockton University of Sunderland, PO Box 299, Sunderland, SR6 0YN, UK
Jakob Nielsen Nielsen Norman Group, 48921 Warm Springs Blvd., Fremont, CA 94539-7767, USA
Jared Spool User Interface Engineering, 242 Neck Road, Bradford, MA 01835, USA
Dennis Wixon Microsoft Corporation, One Microsoft Way, Redmond WA 98052, USA

ABSTRACT Medlock, Wixon et al at Microsoft recommend use of the

Common practice holds that 80% of usability findings are RITE method [3] where changes to a product are made after
discovered after five participants. Recent findings from web having run only 1-3 participants. A survey found that 33 of
testing indicate that a much larger number of participants is 39 respondents had used a similar method of very rapid
required to get results and that independent teams testing iterations and fixes at least once.
the same web-based product do not replicate results. How PANEL POSITIONS
many users are enough for web testing?
Jakob Nielsen
Keywords First, let me clarify that I have always recommended 3-4
Usability test, web, users as the optimal number for most usability studies. The
reason for my more widely known recommendation to plan
OVERVIEW
for 5 users is that this allows for one or even two no-shows
When it comes to web usability testing, the magic number while still getting to observe 3-4 users. Thus, 5 is the
5 is under attack. This number—5 participants will yield magic number during planning, whereas the magic number
80% of the usability findings—derives from research for the actual test is 3-4.
conducted by Nielsen [5], Virzi [8], and Lewis [2].
The standard recommendation to observe 3-4 users refers to
The challenge to this long-held and widely-used practice of studies conducted during a user-centered design process
conducting testing with an average of 5 users began with where an interaction design needs to be debugged for
the results of a comparative usability evaluation [4], in usability. The “discount usability” philosophy explicitly
which nine independent teams conducted usability eval- recognizes that this will not be a perfect study that will
uations of Hotmail, with no two teams finding the same discover everything that’s possible to know about the
set of problems. Of the 300 total problems found, 75% design, but we accept this trade-off in return for having
were reported by only one team. The challenge escalated more iterations in the design process and conserving
with Jared Spool’s report [7] on the results of web testing resources for subsequent evaluations of these iterations.
users purchasing CDs. Thinking it was safe to use the
long-accepted number 5, Spool and his colleagues were The only situation where more than 5 users would be
surprised to see 247 problems identified by 18 different recommended for traditional user interface debugging
users, with major new findings being identified by each would be a design with below-average usability personnel
new user. Kessner et al. [1] found that six professional labs and bad project management. If the test facilitator is not
produced little overlapping findings in evaluating a UI. very observant and if the project is slow to act on the
findings, then it will be most cost-effective to test 10-15
Both Molich et al. [4] and Spool and Schroeder [7] users per iteration.
conclude that web testing requires many more users to get
reliable results. At the same time, Nielsen supports his Gilbert Cockton
original research for “comparable users who will be using Five users can be enough some times. These are extremely
the site in fairly similar ways” [5]. rare. The view that 5 users could be enough was always
Woolrych and Cockton [9] assert that Nielsen’s claim that naive. The statistical arguments behind it were ill informed
“5 Users are Enough” [5] is based on a statistical formula [9]. Where people have tested more than 5 users, it is
[6] that makes unwarranted assumptions about individual absolutely clear that, unless problems are very easy to find,
differences in problem discovery, combined with optimistic not only are more problems found, but also that the profile
setting of values for a key variable. They present evidence of problems as regards frequency and severity changes
that the Landauer-Nielsen formula can fail to calculate an radically with further users. The unpalatable fact is that one
acceptable number of test users even for a small drawing cannot predict in advance how many users are needed. Nor
editor, as the formula only holds for simple problem can one be specific about the risks of testing with 5, 10 or
counts. 15 users in advance of the evidence.
What we do understand are the variables that influence
Copyright is held by the author/owner(s). problem yield, for example, test user diversity, test
Copyright
CHI 2003, isApril
held5–10,
by the2003,
author/owner(s).
Ft. Lauderdale, Florida, USA.
CHI 2003, April 5-10, 2003, Ft. Lauderdale, Florida, USA.
protocol design, task performance diversity, application
ACM 1-58113-630-7/03/0004.
ACM 1-58113-637-4/03/0004 complexity, design quality (problems are easier to find for

698
698
Panels CHI 2003: NEW HORIZONS

poor designs!), problem reporting procedures and the Jared Spool

usability goals set for a product (no goals, no problems!) We have a crisis. And, this argument is in the dead center
However, we do not yet have a formula into which these of it.
variables can be plugged. When we do, the answer will On one level, the notion that 'N' users is required for
rarely be 5. testing (whether 'N' be 5, 8, or some other reasonably small
We must fully share this reality with clients. Usability is number) is an academic question. Design teams are limited
about risk management. Risks diminish as we test more by both time and resources, so they'll test as many users as
users. The break even on cost-benefit is product specific. they can within their constraints. They'll be done when
For some, one user is enough, for others even 100 will be they are done. For whatever value of 'N' they choose, 'N'
too few. users will always be better than zero.
Carol Barnum But, in the grand scheme, there's a much larger and more
The problem that has arisen from the Molich and Spool insidious problem. Practitioners accept usability testing as
studies is that the apparent conclusion—5 is nowhere near the cornerstone of their user-centered design practices. Yet,
enough for web usability testing—has been widely here we have a panel of very smart, well-respected, world-
disseminated, causing great alarm in the usability renowned experts that can't come to any agreement on the
community. Questions have arisen as to whether it’s worth basic elements of a quality testing protocol.
testing if the results are as invalid as have been claimed by For years, the common belief was that Landauer and
these two studies. Nielsen had it right. They had produced a very simple
Our own web studies, conducted according to the formula which could be graphed in the much cited 'parabola
approaches prescribed by the early researchers, and in close of optimism', as we've come to call it. Because we found
cooperation with the client/sponsor, suggest that very that simple software usability tests fit the curve, we
similar results can be duplicated by different test teams. believed all was well with the world. We knew what to do.
Five is enough for web testing: Understanding how many users for testing is more than
• when the original discount model for testing is followed just an academic calculation. It's the basis of faith in
everything we're trying to do to make the world better. We
• when the results of testing are understood and clearly need to come to agreement and we need to do it quickly.
communicated
REFERENCES
• when there is close cooperation between the
1. Kessner, M., Wood, J, Dillon, R.F., and West, R.L.
client/sponsor and the test team
On the reliability of usability testing. CHI 2001.
• when the results are used for diagnostic purposes and Poster.
team learning
2 . Lewis, J.R. Sample sizes for usability studies:
• when the expected result is insight, not validation. Additional considerations. Human Factors 36, 368-378
Dennis Wixon (1994).
The problem of determining how many users one must test 3. Medlock, M.C., Wixon, D., Terrano, M., Romero, R.,
in order to have a reasonable expectation of uncovering all Fulton, B. (2002). Using the RITE method to improve
the problems that users will have is one that has been the products: a definition and a case study. Proc. Usability
subject of much excellent research and theorizing. Professionals Association (Orlando FL, July 2002).
Unfortunately when considered from the viewpoint of 4. Molich, R., et al. Comparative evaluation of usability
designing real products in the real world it is the WRONG tests. CHI 99 Extended Abstracts, ACM Press, 83-84.
PROBLEM TO STUDY. 5. Nielsen, J. Why you only need to test with 5 users.
The goal of most iterative tests in a commercial world is to Alertbox (2000) www.useit.com/alertbox/2000319.html
produce the best possible product/design in the shortest 6. Nielsen, J., and Landauer, T.K. A Mathematical Model
time at the lowest possible cost with the least risk. of the Finding of Usability Problems, in Proc
Consequently, design and usability teams are well advised INTERCHI ’93, 206-213
to fix interface problems as rapidly as possible and 7. Spool, J., and Schroeder, W. Testing web sites: Five
continue testing and fixing until time runs out or a users is nowhere near enough. CHI 2001 Extended
usability metric is achieved. Such an approach (we call it Abstracts, ACM Press, 285-286.
RITE, Rapid Iterative Testing and Evaluation) [3] 8 . Virzi, R. Refining the test phase of usability
concentrates statistical power where it belongs – verifying evaluation: How many subjects is enough? Human
that fixes actually work for users. Factors 34, 457-468 (1992).
While I agree with Gilbert that we must be honest with our 9. Woolrych, A., and Cockton, G. Why and When Five
clients, it is also critical that we understand their concerns Test Users aren't Enough, in Proc. IHM-HCI
and serve their needs. Conference: Volume 2, 105-108, 2001.

699
699