CHAPTER 4:

Evaluating interface Designs

Designing the User Interface:

Strategies for Effective Human-Computer Interaction

Fifth Edition

Ben Shneiderman & Catherine Plaisant

in collaboration with
Maxine S. Cohen and Steven M. Jacobs
Addison Wesley
is an imprint of

© 2010 Pearson Addison-Wesley. All rights reserved.

 Introduction

Evaluation
• tests usability and functionality of system
• occurs in laboratory, field and/or in
collaboration with users
• evaluates both design and implementation

Evaluation should be considered at all stages in

the design life cycle.
1-2

© 2010 Pearson Addison-Wesley. All rights reserved.

 Goals of Evaluation

• assess extent of system functionality

• assess effect of interface on user
• identify specific problems

1-3

© 2010 Pearson Addison-Wesley. All rights reserved.

 Evaluation Plan
• Experienced designers have the wisdom and
humility to know that extensive testing is a
necessity.

• The determinants of the evaluation plan include:

– stage of design (early, middle, late)
– novelty of project (well defined vs. exploratory)
– number of expected users
– criticality of the interface
– costs
– time available
– experience of the design and evaluation team

1-4

© 2010 Pearson Addison-Wesley. All rights reserved. 4-4

 Introduction
• Usability evaluators are open to non-empirical
methods, such as user sketches, consideration of
design alternatives, and ethnographic studies.
• Recommendations needs to be based on
observational findings
• Tools and techniques are evolving
• Evaluation plans range from 3 days to 2 years
• Costs range from 5-20% of a project
• Usability testing has become an established and
accepted part of the design process
1-5

© 2010 Pearson Addison-Wesley. All rights reserved. 4-5

 Expert Reviews
• Informal demos to colleagues or customers can provide
some useful feedback
• Formal expert reviews have proven to be more effective

• Expert review methods:

– Heuristic evaluation
– Guidelines review
– Consistency inspection
– Cognitive walkthrough
– Metaphors of human thinking
– Formal usability inspection

1-6

© 2010 Pearson Addison-Wesley. All rights reserved. 4-6

 Expert Reviews
• Expert reviews can be scheduled at several points in the
development process

• Different experts tend to find different problems in an

interface, so 3-5 expert reviewers recommended

• The dangers with expert reviews are that the experts may
not have an adequate understanding of the task domain
or user communities.

• Even experienced expert reviewers have great difficulty

knowing how typical users, especially first-time users will
really behave.

1-7

© 2010 Pearson Addison-Wesley. All rights reserved. 4-7

 Step-by-Step Usability Guide
from http://usability.gov/

1-8

© 2010 Pearson Addison-Wesley. All rights reserved. 4-8

 Usability Testing and
Laboratories

1-9

© 2010 Pearson Addison-Wesley. All rights reserved. 4-9

 Usability Testing and
Laboratories
• Emergence of usability testing and laboratories since early
1980s

• Sped up many projects and produced dramatic cost

savings.

• A typical modest usability lab would have two 10 by 10 foot

areas
– one for the participants to do their work
– and another, separated by a half-silvered mirror, for the testers and
observers

• Participants should represent the intended user

communities, with attention to
– background in computing, experience with the task, motivation, education,
and ability with the natural language used in the interface. 1-10

© 2010 Pearson Addison-Wesley. All rights reserved. 4-10

 Usability Testing and
Laboratories
• Participation should be voluntary, and informed consent should be
obtained.
• Professional practice is to ask all subjects to read and sign a
statement like this one:
– I have freely volunteered to participate in this
experiment.
– I have been informed in advance what my task(s) will
be and what procedures will be followed.
– I have been given the opportunity to ask questions,
and have had my questions answered to my
satisfaction.
– I am aware that I have the right to withdraw consent
and to discontinue participation at any time, without
prejudice to my future treatment.
– My signature below may be taken as affirmation of all
the above statements; it was given prior to my
participation in this study.
1-11

© 2010 Pearson Addison-Wesley. All rights reserved. 4-11

 Usability Testing and
Laboratories
• Videotaping participants valuable for later review and for showing
designers or managers the problems that users encounter.
– Use caution in order to not interfere with participants
– Invite users to think aloud as they are performing the task.

• Many variant forms of usability testing have been tried:

– Paper mockups
– Discount usability testing
– Competitive usability testing
– Universal usability testing
– Field test and portable labs
– Remote usability testing
– Can-you-break-this tests

1-12
4-12
© 2010 Pearson Addison-Wesley. All rights reserved.
 1-13

© 2010 Pearson Addison-Wesley. All rights reserved.

 Usability Testing and
Laboratories

In this eye-tracking setup, the participant wears a helmet that monitors

and records where on the screen the participant is looking 1-14
4-14
© 2010 Pearson Addison-Wesley. All rights reserved.
 Observational Methods
• paper and pencil
– cheap
– limited to writing speed
• audio
– good for think aloud
– difficult to match with other protocols
• video
– accurate and realistic
– needs special equipment
– obtrusive

Transcription of audio and video difficult/requires skill.

1-15

© 2010 Pearson Addison-Wesley. All rights reserved.

 Observational Methods (cont.)
• computer logging
– automatic and unobtrusive
– large amounts of data difficult to analyze
• user notebooks
– coarse level and subjective
– useful insights
– good for longitudinal studies

• Mixed use in practice.

• Some automatic support tools available
1-16

© 2010 Pearson Addison-Wesley. All rights reserved.

 Survey Instruments
• Written user surveys are
– informal
– subjective
– familiar
– cheap
– good complement to usability tests and expert reviews.

• Keys to successful surveys

– Clear goals in advance
– Development of focused items that help attain the
goals.

• Users asked for subjective impressions about aspects of

the interface such as the representation of:
– task domain objects and actions
– syntax of inputs and design of displays. 1-17

© 2010 Pearson Addison-Wesley. All rights reserved. 4-17

 Survey Instruments
• Other goals would be to ascertain
– users background (age, gender, origins, education, income)
– experience with computers (specific applications or software
packages, length of time, depth of knowledge)
– job responsibilities (decision-making influence, managerial
roles, motivation)
– personality style (introvert vs. extrovert, risk taking vs. risk
aversive, early vs. late adopter, systematic vs. opportunistic)
– reasons for not using an interface (inadequate services, too
complex, too slow)
– familiarity with features (printing, macros, shortcuts, tutorials)
– their feeling state after using an interface (confused vs. clear,
frustrated vs. in-control, bored vs. excited).
1-18
4-18
© 2010 Pearson Addison-Wesley. All rights reserved.
 Surveys
• Online surveys avoid cost of printing and the extra
effort needed for distribution and collection of paper
forms.
• Many prefer to answer a brief survey displayed on a
screen, instead of filling in and returning a printed
form
• A survey example is the Questionnaire for User
Interaction Satisfaction (QUIS).
– http://lap.umd.edu/quis/

1-19
4-19
© 2010 Pearson Addison-Wesley. All rights reserved.
 Styles of question
• general - establish background of user

• open-ended - ‘Can you suggest improvements to interface?’

• scalar

• multi-choice
– How do you most often get help with the system? Choose one.
❑ online manual
❑ contextual help
❑ command prompt
❑ ask a colleague

• ranked – place a list of items in order 1-20

© 2010 Pearson Addison-Wesley. All rights reserved.

 Acceptance Test
• For large projects, the customer or manager usually
sets objective and measurable goals for hardware and
software performance.

• If the completed product fails to meet these acceptance

criteria, the system must be reworked until success is
demonstrated.

• Rather than the vague and misleading criterion of "user

friendly," measurable criteria for the user interface can
be established for the following:
– Time to learn specific functions
– Speed of task performance
– Rate of errors by users
– Human retention of commands over time
– Subjective user satisfaction 1-21

© 2010 Pearson Addison-Wesley. All rights reserved. 4-21

 Evaluation During Active Use

• Successful active use requires constant attention from

dedicated managers, user-services personnel, and
maintenance staff.

• Perfection is not attainable, but percentage

improvements are possible.

• Interviews and focus group discussions

– Interviews productive because the interviewer can
pursue specific issues of concern.
– Group discussions ascertain the universality of
comments.

1-23

© 2010 Pearson Addison-Wesley. All rights reserved. 4-23

 Evaluation During Active Use
• Continuous user-performance data logging
– Collect data about
– The patterns of system usage
– Speed of user performance
– Rate of errors
– Frequency of request for online assistance
– A major benefit is guidance to system maintainers in optimizing
performance and reducing costs for all participants.

• Online or telephone consultants, e-mail, and online

suggestion boxes
– Users feel reassured if human assistance available
– On some systems, consultants can monitor the user's computer
and see the same displays

1-24

© 2010 Pearson Addison-Wesley. All rights reserved. 4-24

 Evaluation During Active Use

• Online suggestion box or e-mail trouble reporting

• Discussion groups, wiki’s and newsgroups

– Permit postings of open messages and questions
– Some are independent, e.g. America Online and Yahoo!
– Topic list
– Sometimes moderators
– Social systems
– Comments and suggestions should be encouraged.

1-25

© 2010 Pearson Addison-Wesley. All rights reserved. 4-25

 Evaluation During Active Use

Bug report using Google’s Chrome browser (http://www.google.com/chrome/)

1-26

© 2010 Pearson Addison-Wesley. All rights reserved. 4-26

 Controlled Psychologically-
oriented Experiments

• Scientific and engineering progress is often stimulated by

improved techniques for precise measurement.

1-27

© 2010 Pearson Addison-Wesley. All rights reserved. 4-27

 Controlled Experiments
• Outline of scientific method as applied to HCI:

– Deal with a practical problem and consider the theoretical

framework
– State a lucid and testable hypothesis
– Identify a small number of independent variables (characteristics)
to manipulate to produce different conditions
• eg interface style, number of menu items
– Choose the dependent variables that will be measured
• eg time taken, number of errors
– Judiciously select subjects and carefully or randomly assign
subjects to groups
– Control for biasing factors (non-representative sample of subjects
or selection of tasks, inconsistent testing procedures)
– Apply statistical methods to data analysis
– Resolve the practical problem, refine the theory, and give advice
to future researchers
1-28

© 2010 Pearson Addison-Wesley. All rights reserved. 4-28

 Controlled Experiments Evaluation

• Experimental design
– Within groups design - each subject
performs experiment under each condition.
Transfer of learning possible but less costly
and less likely to suffer from user variation.

– Between groups design - each subject

performs under only one condition. No
transfer of learning but more users required
and variation can bias results.
1-29

© 2010 Pearson Addison-Wesley. All rights reserved.

 Statistical Analysis: Types of Tests
• parametric
– assume normal distribution
– robust
– powerful
• non-parametric
– do not assume normal distribution
– less powerful
– more reliable
• contingency table
– classify data by discrete attributes and count number
of data items in each group 1-30

© 2010 Pearson Addison-Wesley. All rights reserved.

 Choosing an Evaluation Method

• Factors to consider
– when in cycle is evaluation carried out? design vs
implementation
– what style of evaluation is required? laboratory vs field
– how objective should the technique be? subjective vs
objective
– what type of measures are required? qualitative vs
quantitative
– what level of information is required? high level vs low
level
– what level of interference? obtrusive vs unobtrusive
– what resources are available? time, subjects, equipment,
expertise

1-32

© 2010 Pearson Addison-Wesley. All rights reserved.